Small-sample precision of ROC-related estimates

• DOI: 10.1093/bioinformatics/btq037
• Publication Date: 2010-02-04T01:55:22Z
• Authors: Blaise Hanczar; Jianping Hua; Chao Sima; John Weinstein; Michael Bittner; Edward R. Dougherty
• Keywords: Resampling; False positive rate; Discriminant function analysis

Abstract Motivation: The receiver operator characteristic (ROC) curves are commonly used in biomedical applications to judge the performance of a discriminant across varying decision thresholds. estimated ROC curve depends on true positive rate (TPR) and false (FPR), with key metric being area under (AUC). With small samples these rates need be from training data, so natural question arises: How well do estimates AUC, TPR FPR compare metrics? Results: Through simulation study using data models analysis real microarray we show that (i) for root mean square differences metrics considerable; (ii) even large samples, there is only weak correlation between metrics; (iii) generally, regression metric. For classification rules, consider linear analysis, support vector machine (SVM) radial basis function SVM. error estimation, resubstitution, three kinds cross-validation bootstrap. Using resampling, unreliability some published results. Availability: Companion web site at http://compbio.tgen.org/paper_supp/ROC/roc.html Contact: [e-mail]