Cross-study validation for the assessment of prediction algorithms

• DOI: 10.1093/bioinformatics/btu279
• Publication Date: 2014-06-16T21:55:09Z
• Authors: Christoph Bernau; Markus Riester; Anne-Laure Boules...; Giovanni Parmigiani; Curtis Huttenhower; Levi Waldron; Lorenzo Trippa
• Keywords: Cross-validation; Predictive modelling

Abstract Motivation: Numerous competing algorithms for prediction in high-dimensional settings have been developed the statistical and machine-learning literature. Learning models they generate are typically evaluated on basis of cross-validation error estimates a few exemplary datasets. However, most applications, ultimate goal modeling is to provide accurate predictions independent samples obtained different settings. Cross-validation within datasets may not adequately reflect performance broader application context. Methods: We develop implement systematic approach ‘cross-study validation’, replace or supplement conventional when evaluating illustrate it via simulations collection eight estrogen-receptor positive breast cancer microarray gene-expression datasets, where objective predicting distant metastasis-free survival (DMFS). computed C-index all pairwise combinations training validation evaluate several alternatives summarizing statistics, compare these cross-validation. Results: Our data-driven our with suggest that standard produces inflated discrimination accuracy considered, compared cross-study validation. Furthermore, ranking learning differs, suggesting performing best be suboptimal through Availability: The survHD: Survival High Dimensions package (http://www.bitbucket.org/lwaldron/survhd) will made available Bioconductor. Contact: [e-mail] Supplementary information: data at Bioinformatics online.