External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb

• DOI: 10.1016/j.jclinepi.2021.02.011
• Publication Date: 2021-02-16T06:40:14Z
• Authors: Kym I.E. Snell; Lucinda Archer; Joie Ensor; Laura J. Bonnett; Thomas P.A. Debray; Bob Phillips; Gary S. Collins; Richard D. Riley
• Keywords: Rule of thumb; Sample (material); Model Validation

IntroductionSample size "rules-of-thumb" for external validation of clinical prediction models suggest at least 100 events and non-events. Such blanket guidance is imprecise, not specific to the model or setting. We investigate factors affecting precision performance estimates upon validation, propose a more tailored sample approach.MethodsSimulation logistic regression associated with estimates. Then, explanation illustration simulation-based approach calculate minimum required precisely estimate model's calibration, discrimination utility.ResultsPrecision affected by linear predictor (LP) distribution, in addition number total size. Sample sizes (or even 200) non-events can give imprecise estimates, especially calibration. The calculation accounts LP distribution (mis)calibration sample. Application identifies 2430 participants (531 events) deep vein thrombosis diagnostic model.ConclusionWhere researchers anticipate (eg, based on development sample, pilot study), calculating offers flexibility reliability than rules-of-thumb.