JAWS: Auditing Predictive Uncertainty Under Covariate Shift

• DOI: 10.48550/arxiv.2207.10716
• Publication Date: 2022-01-01
• Authors: Drew Prinster; Anqi Liu; Suchi Saria
• Keywords: Jackknife resampling

We propose \textbf{JAWS}, a series of wrapper methods for distribution-free uncertainty quantification tasks under covariate shift, centered on the core method \textbf{JAW}, \textbf{JA}ckknife+ \textbf{W}eighted with data-dependent likelihood-ratio weights. JAWS also includes computationally efficient \textbf{A}pproximations JAW using higher-order influence functions: \textbf{JAWA}. Theoretically, we show that relaxes jackknife+'s assumption data exchangeability to achieve same finite-sample coverage guarantee even shift. JAWA further approaches in limit sample size or function order common regularity assumptions. Moreover, general approach repurposing predictive interval-generating and their guarantees reverse task: estimating probability prediction is erroneous, based user-specified error criteria such as safe acceptable tolerance threshold around true label. then \textbf{JAW-E} \textbf{JAWA-E} repurposed proposed this \textbf{E}rror assessment task. Practically, outperform state-of-the-art inference baselines variety biased real world sets interval-generation error-assessment auditing tasks.