Evaluation of Natural Language Processing for the Identification of Crohn Disease–Related Variables in Spanish Electronic Health Records: A Validation Study for the PREMONITION-CD Project (Preprint)

• DOI: 10.2196/preprints.30345
• Publication Date: 2022-02-18T15:15:37Z
• Authors: Carmen Montoto; Javier P Gisbert; Iván Guerra; Rocío Plaza; Ramón Pajares Vil...; Luis Moreno Almazán; María Del Carmen ...; Mercedes Domíngue...; Isabel Vera Mendoza; Jesús Aparicio; Vicente Martínez; Ignacio Tagarro; Alonso Fernandez-...; Lea Canales; Sebastian Menke; Fernando Gomollón
• Keywords: linguistic validation; 0301 basic medicine; Original Paper; Computer applications to medicine. Medical informatics; R858-859.7; Crohn disease; Procesamiento de Lenguaje Natural; Registros Electrónicos de Salud; artificial intelligence; Inflammatory Bowel Diseases; 03 medical and health sciences; electronic health records; Enfermedades Inflamatorias del Intestino; 0302 clinical medicine; Crohn Disease; inflammatory bowel disease; Artificial Intelligence; Electronic Health Records; natural language processing; Enfermedad de Crohn; Inteligencia Artificial; Natural Language Processing

BACKGROUND The exploration of clinically relevant information in the free text of electronic health records (EHRs) holds the potential to positively impact clinical practice as well as knowledge regarding Crohn disease (CD), an inflammatory bowel disease that may affect any segment of the gastrointestinal tract. The EHRead technology, a clinical natural language processing (cNLP) system, was designed to detect and extract clinical information from narratives in the clinical notes contained in EHRs.

OBJECTIVE The aim of this study is to validate the performance of the EHRead technology in identifying information of patients with CD.

METHODS We used the EHRead technology to explore and extract CD-related clinical information from EHRs. To validate this tool, we compared the output of the EHRead technology with a manually curated gold standard to assess the quality of our cNLP system in detecting records containing any reference to CD and its related variables.

RESULTS The validation metrics for the main variable (CD) were a precision of 0.88, a recall of 0.98, and an F1 score of 0.93. Regarding the secondary variables, we obtained a precision of 0.91, a recall of 0.71, and an F1 score of 0.80 for CD flare, while for the variable vedolizumab (treatment), a precision, recall, and F1 score of 0.86, 0.94, and 0.90 were obtained, respectively.

CONCLUSIONS This evaluation demonstrates the ability of the EHRead technology to identify patients with CD and their related variables from the free text of EHRs. To the best of our knowledge, this study is the first to use a cNLP system for the identification of CD in EHRs written in Spanish.