On the loss landscape of a class of deep neural networks with no bad local valleys

• DOI: 10.48550/arxiv.1809.10749
• Publication Date: 2018-01-01
• Authors: Nguyen, Quynh; Mukkamala, Mahesh...; Hein, Matthias
• Keywords: FOS: Computer and information sciences; Computer Science - Machine Learning; Artificial Intelligence (cs.AI); Computer Science - Artificial Intelligence; Statistics - Machine Learning; Computer Vision and Pattern Recognition (cs.CV); Computer Science - Computer Vision and Pattern Recognition; Machine Learning (stat.ML); 01 natural sciences; 0105 earth and related environmental sciences; Machine Learning (cs.LG)

We identify a class of over-parameterized deep neural networks with standard activation functions and cross-entropy loss which provably have no bad local valley, in the sense that from any point in parameter space there exists a continuous path on which the cross-entropy loss is non-increasing and gets arbitrarily close to zero. This implies that these networks have no sub-optimal strict local minima.<br/>Accepted at ICLR 2019<br/>