The NLP Sandbox: an efficient model-to-data system to enable federated and unbiased evaluation of clinical NLP models

• DOI: 10.48550/arxiv.2206.14181
• Publication Date: 2022-01-01
• Authors: Yan Yao; Thomas Yu; Kathleen Muenzen; Sijia Liu; Connor Boyle; George Koslowski; Jiaxin Zheng; Nicholas J Dobbins; Clement Essien; Hongfang Liu; Larsson Omberg; Meliha Yestigen; Bradley Taylor; James A. Eddy; Justin Guinney; Sean D. Mooney; Thomas Schaffter
• Keywords: Sandbox (software development); Identification

Objective The evaluation of natural language processing (NLP) models for clinical text de-identification relies on the availability notes, which is often restricted due to privacy concerns. NLP Sandbox an approach alleviating lack data and frameworks by adopting a federated, model-to-data approach. This enables unbiased federated model without need sharing sensitive from multiple institutions. Materials Methods We leveraged Synapse collaborative framework, containerization software, OpenAPI generator build (nlpsandbox.io). evaluated two state-of-the-art focused annotation models, Philter NeuroNER, using three further validated performance external validation site. Results demonstrated usefulness through evaluation. developer was able incorporate their into template provide user experience feedback. Discussion feasibility conduct multi-site data. Standardized schemas enable smooth transfer implementation. To generalize Sandbox, work required part owners developers develop suitable standardized adapt or fit schemas. Conclusions lowers barrier utilizing facilitates multi-site, models.