Gradient Networks

• DOI: 10.48550/arxiv.2404.07361
• Publication Date: 2025-01-01
• Authors: Shreyas Chaudhari; Srinivasa Pranav; José M.F. Moura
• Keywords: Signal Processing (eess.SP); FOS: Computer and information sciences; Computer Science - Machine Learning; Optimization and Control (math.OC); FOS: Electrical engineering, electronic engineering, information engineering; FOS: Mathematics; Computer Science - Neural and Evolutionary Computing; Neural and Evolutionary Computing (cs.NE); Electrical Engineering and Systems Science - Signal Processing; Mathematics - Optimization and Control; Machine Learning (cs.LG)

Directly parameterizing and learning gradients of functions has widespread significance, with specific applications in inverse problems, generative modeling, and optimal transport. This paper introduces gradient networks (GradNets): novel neural network architectures that parameterize gradients of various function classes. GradNets exhibit specialized architectural constraints that ensure correspondence to gradient functions. We provide a comprehensive GradNet design framework that includes methods for transforming GradNets into monotone gradient networks (mGradNets), which are guaranteed to represent gradients of convex functions. Our results establish that our proposed GradNet (and mGradNet) universally approximate the gradients of (convex) functions. Furthermore, these networks can be customized to correspond to specific spaces of potential functions, including transformed sums of (convex) ridge functions. Our analysis leads to two distinct GradNet architectures, GradNet-C and GradNet-M, and we describe the corresponding monotone versions, mGradNet-C and mGradNet-M. Our empirical results demonstrate that these architectures provide efficient parameterizations and outperform existing methods by up to 15 dB in gradient field tasks and by up to 11 dB in Hamiltonian dynamics learning tasks.<br/>Published in IEEE Transactions on Signal Processing<br/>