Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift

• DOI: 10.48550/arxiv.1906.02530
• Publication Date: 2019-01-01
• Authors: Yaniv Ovadia; Emily Fertig; Jie Ren; Zachary Nado; D. Sculley; Sebastian Nowozin; Joshua V. Dillon; Balaji Lakshminar...; Jasper Snoek
• Keywords: Benchmark (surveying); Uncertainty Quantification; Predictive power

Modern machine learning methods including deep have achieved great success in predictive accuracy for supervised tasks, but may still fall short giving useful estimates of their {\em uncertainty}. Quantifying uncertainty is especially critical real-world settings, which often involve input distributions that are shifted from the training distribution due to a variety factors sample bias and non-stationarity. In such well calibrated convey information about when model's output should (or not) be trusted. Many probabilistic methods, Bayesian-and non-Bayesian been proposed literature quantifying uncertainty, our knowledge there has not previously rigorous large-scale empirical comparison these under dataset shift. We present benchmark existing state-of-the-art on classification problems investigate effect shift calibration. find traditional post-hoc calibration does indeed short, as do several other previous methods. However, some marginalize over models give surprisingly strong results across broad spectrum tasks.