Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction is a user-friendly paradigm for creating statistically rigorous sets/intervals the predictions of such models. Critically, sets valid distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or assumptions. One can use conformal with...

Large Language Models (LLMs) have unlocked new capabilities and applications; however, evaluating the alignment with human preferences still poses significant challenges. To address this issue, we introduce Chatbot Arena, an open platform for LLMs based on preferences. Our methodology employs a pairwise comparison approach leverages input from diverse user base through crowdsourcing. The has been operational several months, amassing over 240K votes. This paper describes platform, analyzes...

The cameras in modern gaze-tracking systems suffer from fundamental bandwidth and power limitations, constraining data acquisition speed to 300 Hz realistically. This obstructs the use of mobile eye trackers perform, e.g., low latency predictive rendering, or study quick subtle motions like microsaccades using head-mounted devices wild. Here, we propose a hybrid frame-event-based near-eye gaze tracking system offering update rates beyond 10,000 with an accuracy that matches high-end...

Prediction-powered inference is a framework for performing valid statistical when an experimental dataset supplemented with predictions from machine-learning system. The yields simple algorithms computing provably confidence intervals quantities such as means, quantiles, and linear logistic regression coefficients without making any assumptions about the algorithm that supplies predictions. Furthermore, more accurate translate to smaller intervals. could enable researchers draw...

Abstract Molecular structure prediction and homology detection offer promising paths to discovering protein function evolutionary relationships. However, current approaches lack statistical reliability assurances, limiting their practical utility for selecting proteins further experimental in-silico characterization. To address this challenge, we introduce a statistically principled approach search leveraging principles from conformal prediction, offering framework that ensures guarantees...

Immersive computer graphics systems strive to generate perceptually realistic user experiences. Current-generation virtual reality (VR) displays are successful in accurately rendering many important effects, including perspective, disparity, motion parallax, and other depth cues. In this article, we introduce ocular parallax rendering, a technology that renders small amounts of gaze-contingent capable improving perception realism VR. Ocular describes the depth-dependent image shifts on...

While improving prediction accuracy has been the focus of machine learning in recent years, this alone does not suffice for reliable decision-making. Deploying systems consequential settings also requires calibrating and communicating uncertainty predictions. To convey instance-wise tasks, we show how to generate set-valued predictions from a black-box predictor that controls expected loss on future test points at user-specified level. Our approach provides explicit finite-sample guarantees...

Many applications of machine learning methods involve an iterative protocol in which data are collected, a model is trained, and then outputs that used to choose what consider next. For example, one data-driven approach for designing proteins train regression predict the fitness protein sequences, use it propose new sequences believed exhibit greater than observed training data. Since validating designed wet lab typically costly, important quantify uncertainty model's predictions. This...

We present a new perspective on online learning that we refer to as gradient equilibrium: sequence of iterates achieves equilibrium if the average gradients losses along converges zero. In general, this condition is not implied by nor implies sublinear regret. It turns out achievable standard methods such descent and mirror with constant step sizes (rather than decaying sizes, usually required for no regret). Further, show through examples, translates into an interpretable meaningful...

Evaluating in-the-wild coding capabilities of large language models (LLMs) is a challenging endeavor with no clear solution. We introduce Copilot Arena, platform to collect user preferences for code generation through native integration into developer's working environment. Arena comprises novel interface comparing pairs model outputs, sampling strategy optimized reduce latency, and prompting scheme enable completion functionality. has served over 4.5 million suggestions from 10 collected...

Large language model (LLM) evaluations typically rely on aggregated metrics like accuracy or human preference, averaging across users and prompts. This obscures user- prompt-specific variations in performance. To address this, we propose Prompt-to-Leaderboard (P2L), a method that produces leaderboards specific to prompt. The core idea is train an LLM taking natural prompts as input output vector of Bradley-Terry coefficients which are then used predict the preference vote. resulting...

Abstract Patients diagnosed with Retinitis Pigmentosa (RP) show, in the advanced stage of disease, severely restricted peripheral vision causing poor mobility and decline quality life. This loss causes difficulty identifying obstacles their relative distances. Thus, RP patients use aids such as canes to navigate, especially dark environments. A number high-tech visual using virtual reality (VR) sensory substitution have been developed support or supplant traditional aids. These not achieved...

We extend conformal prediction to control the expected value of any monotone loss function. The algorithm generalizes split together with its coverage guarantee. Like prediction, risk procedure is tight up an $\mathcal{O}(1/n)$ factor. also introduce extensions idea distribution shift, quantile control, multiple and adversarial expectations U-statistics. Worked examples from computer vision natural language processing demonstrate usage our bound false negative rate, graph distance,...

Abstract The relative case fatality rates (CFRs) between groups and countries are key measures of risk that guide policy decisions regarding scarce medical resource allocation during the ongoing COVID-19 pandemic. In middle an active outbreak when surveillance data is primary source information, estimating these quantities involves compensating for competing biases in time series deaths, cases, recoveries. These include time- severity-dependent reporting cases as well lags observed patient...

We introduce a framework for calibrating machine learning models so that their predictions satisfy explicit, finite-sample statistical guarantees. Our calibration algorithms work with any underlying model and (unknown) data-generating distribution do not require refitting. The addresses, among other examples, false discovery rate control in multi-label classification, intersection-over-union instance segmentation, the simultaneous of type-1 error outlier detection confidence set coverage...

Decision-making pipelines are generally characterized by tradeoffs among various risk functions. It is often desirable to manage such in a data-adaptive manner. As we demonstrate, if this done naively, state-of-the art uncertainty quantification methods can lead significant violations of putative guarantees. To address issue, develop that permit valid control when threshold and tradeoff parameters chosen adaptively. Our methodology supports monotone nearly-monotone risks, but otherwise makes...