Exascale Deep Learning for Scientific Inverse Problems

• DOI: 10.48550/arxiv.1909.11150
• Publication Date: 2019-01-01
• Authors: Nouamane Laanait; Joshua Romero; Junqi Yin; M. Todd Young; Sean Treichler; Vitalii Starchenko; Albina Y. Borisevich; A. E. Sergeev; Michael A. Matheson
• Keywords: FOS: Computer and information sciences; Computer Science - Machine Learning; Condensed Matter - Materials Science; Materials Science (cond-mat.mtrl-sci); FOS: Physical sciences; Machine Learning (stat.ML); 02 engineering and technology; Computational Physics (physics.comp-ph); Machine Learning (cs.LG); Computer Science - Distributed, Parallel, and Cluster Computing; Statistics - Machine Learning; 0202 electrical engineering, electronic engineering, information engineering; Distributed, Parallel, and Cluster Computing (cs.DC); Physics - Computational Physics

We introduce novel communication strategies in synchronous distributed Deep Learning consisting of decentralized gradient reduction orchestration and computational graph-aware grouping tensors. These new techniques produce an optimal overlap between computation result near-linear scaling (0.93) training up to 27,600 NVIDIA V100 GPUs on the Summit Supercomputer. demonstrate our context a Fully Convolutional Neural Network approximate solution longstanding scientific inverse problem materials imaging. The efficient dataset size 0.5 PB, produces model capable atomically-accurate reconstruction materials, process reaching peak performance 2.15(4) EFLOPS$_{16}$.