A Proximal Stochastic Quasi-Newton Algorithm

• DOI: 10.48550/arxiv.1602.00223
• Publication Date: 2016-01-01
• Authors: Luo, Luo; Chen, Zihao; Zhang, Zhihua; Li, Wu-Jun
• Keywords: FOS: Computer and information sciences; Computer Science - Machine Learning; Statistics - Machine Learning; 0211 other engineering and technologies; Machine Learning (stat.ML); 02 engineering and technology; Machine Learning (cs.LG)

In this paper, we discuss the problem of minimizing the sum of two convex functions: a smooth function plus a non-smooth function. Further, the smooth part can be expressed by the average of a large number of smooth component functions, and the non-smooth part is equipped with a simple proximal mapping. We propose a proximal stochastic second-order method, which is efficient and scalable. It incorporates the Hessian in the smooth part of the function and exploits multistage scheme to reduce the variance of the stochastic gradient. We prove that our method can achieve linear rate of convergence.