WONDER: Weighted one-shot distributed ridge regression in high dimensions

• DOI: 10.48550/arxiv.1903.09321
• Publication Date: 2019-01-01
• Authors: Edgar Dobriban; Sheng Yue
• Keywords: Wonder

In many areas, practitioners need to analyze large datasets that challenge conventional single-machine computing. To scale up data analysis, distributed and parallel computing approaches are increasingly needed. Here we study a fundamental highly important problem in this area: How do ridge regression environment? Ridge is an extremely popular method for supervised learning, has several optimality properties, thus it study. We one-shot methods construct weighted combinations of estimators computed on each machine. By analyzing the mean squared error high dimensional random-effects model where predictor small effect, discover new phenomena. 1. Infinite-worker limit: The estimator works well very numbers machines, phenomenon call "infinite-worker limit". 2. Optimal weights: optimal weights combining local sum more than unity, due downward bias ridge. Thus, all averaging suboptimal. also propose Weighted ONe-shot DistributEd (WONDER) algorithm. test WONDER simulation studies using Million Song Dataset as example. There can save at least 100x computation time, while nearly preserving accuracy.