A Primer on Coordinate Descent Algorithms

• DOI: 10.48550/arxiv.1610.00040
• Publication Date: 2016-01-01
• Authors: Shi, Hao-Jun Michael; Tu, Shenyinying; Xu, Yangyang; Yin, Wotao
• Keywords: FOS: Computer and information sciences; Statistics - Machine Learning; Optimization and Control (math.OC); 0211 other engineering and technologies; FOS: Mathematics; Machine Learning (stat.ML); 02 engineering and technology; Mathematics - Optimization and Control

This monograph presents a class of algorithms called coordinate descent algorithms for mathematicians, statisticians, and engineers outside the field of optimization. This particular class of algorithms has recently gained popularity due to their effectiveness in solving large-scale optimization problems in machine learning, compressed sensing, image processing, and computational statistics. Coordinate descent algorithms solve optimization problems by successively minimizing along each coordinate or coordinate hyperplane, which is ideal for parallelized and distributed computing. Avoiding detailed technicalities and proofs, this monograph gives relevant theory and examples for practitioners to effectively apply coordinate descent to modern problems in data science and engineering.