Special Issue on CDS Failures: Performance Degradation between Development and Deployment of a Predictive Model for Central-Line Associated Blood Stream Infections in Hospitalized Children

• DOI: 10.1055/a-2605-1847
• Publication Date: 2025-05-13T00:00:47Z
• Authors: Jonathan M Beus; Mark Mai; Nikolay P Braykov; Swaminathan Kanda...; Edwin Ray; D Brad Cundiff; Paulette Djachechi; Sarah A Thompson; Azade Tabaie; Ryan Birmingham; Rishi Kamaleswaran; Evan Orenstein
• Keywords: Central line; Predictive modelling; Predictive Analytics

Background: Central line-associated bloodstream infections (CLABSIs) are associated with substantial pediatric morbidity and mortality. The capacity to predict which children central lines at greatest risk of CLABSI could inform surveillance prevention efforts. Our team previously published in silico predictive models for CLABSI. Objective: To prospectively implement a model achieve adequate performance offline validation implementation clinical practice. Methods: most performant were deep learning requiring pre-processing many features into 8-hour windows including the current day up 56 days prior admission. replicate this pre-processing, we created novel infrastructure (1) organize current-day data all relevant (2) create staged historical store those same application programming interfaces connect two. We compared these scores next 48 hours two labels, one based on manual review positive blood cultures another culture receipt least 4 new IV antibiotics. Results: area under receiver-operating characteristic (AUROC) fell from 0.97 retrospective <0.60 despite multiple iterations troubleshooting. Primary root causes included train/serve skew, feature leakage, overfitting. Hypothesized secondary drivers complex specification, poor governance inadequate testing, challenging translation between real-time models, limited monitoring logging troubleshooting, suboptimal handoff development deployment teams. Conclusion: bridge gap requires early close coordination governance, science, informatics, engineers. Balancing feasibility can accelerate adoption decision support systems.