Understanding GNN Computational Graph: A Coordinated Computation, IO, and Memory Perspective

• DOI: 10.48550/arxiv.2110.09524
• Publication Date: 2021-01-01
• Authors: Hengrui Zhang; Zhongming Yu; Guohao Dai; Guyue Huang; Yufei Ding; Yuan Xie; Yu Wang
• Keywords: Speedup

Graph Neural Networks (GNNs) have been widely used in various domains, and GNNs with sophisticated computational graph lead to higher latency larger memory consumption. Optimizing the GNN suffers from: (1) Redundant neural operator computation. The same data are propagated through structure perform operation multiple times GNNs, leading redundant computation which accounts for 92.4% of total operators. (2) Inconsistent thread mapping. Efficient mapping schemes vertex-centric edge-centric operators different. This inconsistency prohibits fusion reduce IO. (3) Excessive intermediate data. For training is usually performed concurrently inference, must be stored backward pass, consuming 91.9% requirement. To tackle these challenges, we propose following designs optimize from a novel coordinated computation, IO, perspective: Propagation-postponed reorganization. We reorganize operations before propagation, thus eliminated. Unified fusion. unified scheme both vertex- enable Intermediate recomputation. recomputed during pass Extensive experimental results on three typical models show that, achieve up 2.75x end-to-end speedup, 6.89x less 7.73x consumption over state-of-the-art frameworks.