A stochastic optimization approach to train non-linear neural networks with a higher-order variation regularization

• DOI: 10.48550/arxiv.2308.02293
• Publication Date: 2023-01-01
• Authors: Akifumi Okuno
• Keywords: Overfitting; Regularization; Automatic differentiation; Stochastic Gradient Descent; Parametric model

While highly expressive parametric models including deep neural networks have an advantage to model complicated concepts, training such non-linear is known yield a high risk of notorious overfitting. To address this issue, study considers $(k,q)$th order variation regularization ($(k,q)$-VR), which defined as the $q$th-powered integral absolute $k$th derivative be trained; penalizing $(k,q)$-VR expected smoother function, avoid Particularly, encompasses conventional (general-order) total with $q=1$. terms applied general are computationally intractable due integration, provides stochastic optimization algorithm, that can efficiently train without conducting explicit numerical integration. The proposed approach even whose structure arbitrary, it implemented by only simple gradient descent algorithm and automatic differentiation. Our experiments demonstrate trained more ``resilient'' than those parameter regularization. also extended physics-informed (PINNs).