Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning

• DOI: 10.48550/arxiv.2003.07329
• Publication Date: 2020-01-01
• Authors: Jize Zhang; Bhavya Kailkhura; T. Yong-Jin Han
• Keywords: Ensemble Learning; Kernel (algebra)

This paper studies the problem of post-hoc calibration machine learning classifiers. We introduce following desiderata for uncertainty calibration: (a) accuracy-preserving, (b) data-efficient, and (c) high expressive power. show that none existing methods satisfy all three requirements, demonstrate how Mix-n-Match strategies (i.e., ensemble composition) can help achieve remarkably better data-efficiency power while provably maintaining classification accuracy original classifier. are generic in sense they be used to improve performance any off-the-shelf calibrator. also reveal potential issues standard evaluation practices. Popular approaches (e.g., histogram-based expected error (ECE)) may provide misleading results especially small-data regime. Therefore, we propose an alternative data-efficient kernel density-based estimator a reliable prove its asymptotically unbiasedness consistency. Our outperform state-of-the-art solutions on both as well tasks most experimental settings. codes available at https://github.com/zhang64-llnl/Mix-n-Match-Calibration.