Active label cleaning for improved dataset quality under resource constraints

• DOI: 10.1038/s41467-022-28818-3
• Publication Date: 2022-03-04T13:37:47Z
• Authors: Mélanie Bernhardt; Daniel C. Castro; Ryutaro Tanno; Anton Schwaighofer; Kerem C. Tezcan; Miguel Monteiro; Shruthi Bannur; Matthew P. Lungren; Aditya Nori; Ben Glocker; Javier Alvarez-Valle; Ozan Oktay
• Keywords: Benchmark (surveying)

Imperfections in data annotation, known as label noise, are detrimental to the training of machine learning models and have a confounding effect on assessment model performance. Nevertheless, employing experts remove noise by fully re-annotating large datasets is infeasible resource-constrained settings, such healthcare. This work advocates for data-driven approach prioritising samples re-annotation-which we term "active cleaning". We propose rank instances according estimated correctness labelling difficulty each sample, introduce simulation framework evaluate relabelling efficacy. Our experiments natural images specifically-devised medical imaging benchmark show that cleaning noisy labels mitigates their negative impact training, evaluation, selection. Crucially, proposed enables correcting up 4 × more effectively than typical random selection realistic conditions, making better use experts' valuable time improving dataset quality.