All models are wrong, some are useful: Model Selection with Limited Labels

• DOI: 10.48550/arxiv.2410.13609
• Publication Date: 2024-10-17
• Authors: Patrik Okanovic; Andreas Kirsch; J. C. Kasper; Torsten Hoefler; Andreas Krause; Nezihe Merve Gürel
• Keywords: FOS: Computer and information sciences; Computer Science - Machine Learning; Machine Learning (cs.LG)

With the multitude of pretrained models available thanks to advancements in large-scale supervised and self-supervised learning, choosing right model is becoming increasingly pivotal machine learning lifecycle. However, much like training process, best off-the-shelf for raw, unlabeled data a labor-intensive task. To overcome this, we introduce MODEL SELECTOR, framework label-efficient selection classifiers. Given pool target data, SELECTOR samples small subset highly informative examples labeling, order efficiently identify deployment on this dataset. Through extensive experiments, demonstrate that drastically reduces need labeled while consistently picking or near-best performing model. Across 18 collections 16 different datasets, comprising over 1,500 models, labeling cost by up 94.15% compared strongest baseline. Our results further highlight robustness selection, as it 72.41% when selecting model, whose accuracy only within 1%