Information-Theoretic Local Minima Characterization and Regularization

• DOI: 10.48550/arxiv.1911.08192
• Publication Date: 2019-01-01
• Authors: Zhiwei Jia; Hao Su
• Keywords: Maxima and minima; Regularization; Deep Neural Networks

Recent advances in deep learning theory have evoked the study of generalizability across different local minima neural networks (DNNs). While current work focused on either discovering properties good or developing regularization techniques to induce minima, no approach exists that can tackle both problems. We achieve these two goals successfully a unified manner. Specifically, based observed Fisher information we propose metric strongly indicative and effectively applied as practical regularizer. provide theoretical analysis including generalization bound empirically demonstrate success our capturing improving DNNs. Experiments are performed CIFAR-10, CIFAR-100 ImageNet for various network architectures.