Thoughts on Massively Scalable Gaussian Processes

• DOI: 10.48550/arxiv.1511.01870
• Publication Date: 2015-01-01
• Authors: Andrew Gordon Wilson; Christoph Dann; Hannes Nickisch
• Keywords: Methodology (stat.ME); FOS: Computer and information sciences; Computer Science - Machine Learning; 0209 industrial biotechnology; Artificial Intelligence (cs.AI); Computer Science - Artificial Intelligence; Statistics - Machine Learning; 0202 electrical engineering, electronic engineering, information engineering; Machine Learning (stat.ML); 02 engineering and technology; Statistics - Methodology; Machine Learning (cs.LG)

We introduce a framework and early results for massively scalable Gaussian processes (MSGP), significantly extending the KISS-GP approach of Wilson Nickisch (2015). The MSGP enables use (GPs) on billions datapoints, without requiring distributed inference, or severe assumptions. In particular, reduces standard $O(n^3)$ complexity GP learning inference to $O(n)$, $O(n^2)$ per test point prediction $O(1)$. involves 1) decomposing covariance matrices as Kronecker products Toeplitz approximated by circulant matrices. This multi-level approximation allows one unify orthogonal computational benefits fast approaches, is faster than either in isolation; 2) local kernel interpolation inducing points allow arbitrarily located data inputs, $O(1)$ time predictions; 3) exploiting block-Toeplitz Toeplitz-block structure (BTTB), which when multidimensional not present; 4) projections input space flexibly model correlated inputs high dimensional data. ability handle many ($m \approx n$) near-exact accuracy large scale learning.