Physician Detection of Clinical Harm in Machine Translation: Quality Estimation Aids in Reliance and Backtranslation Identifies Critical Errors

• DOI: 10.48550/arxiv.2310.16924
• Publication Date: 2023-01-01
• Authors: Nikita Mehandru; Sweta Agrawal; Yimin Xiao; Elaine C. Khoong; Ge Gao; Marine Carpuat; Niloufar Salehi
• Keywords: FOS: Computer and information sciences; Computer Science - Computation and Language; Computer Science - Human-Computer Interaction; Computation and Language (cs.CL); 3. Good health; Human-Computer Interaction (cs.HC)

A major challenge in the practical use of Machine Translation (MT) is that users lack guidance to make informed decisions about when rely on outputs. Progress quality estimation research provides techniques automatically assess MT quality, but these have primarily been evaluated vitro by comparison against human judgments outside a specific context use. This paper evaluates feedback vivo with study simulating decision-making high-stakes medical settings. Using Emergency Department discharge instructions, we how interventions based versus backtranslation assist physicians deciding whether show outputs patient. We find improves appropriate reliance MT, helps detect more clinically harmful errors QE alone often misses.