train sort explain learning to diagnose translation models

• DOI: 10.48550/arxiv.1903.12017
• Publication Date: 2019-01-01
• Authors: Eleftherios Avram...; Robert Schwarzenberg; Sebastian Möller; David Harbecke; Vivien Macketanz
• Keywords: FOS: Computer and information sciences; Computer Science - Machine Learning; Computer Science - Computation and Language; Computation and Language (cs.CL); 01 natural sciences; Machine Learning (cs.LG); 0105 earth and related environmental sciences

NAACL-HLT 2019: Demonstrations<br/>Evaluating translation models is a trade-off between effort and detail. On the one end of the spectrum there are automatic count-based methods such as BLEU, on the other end linguistic evaluations by humans, which arguably are more informative but also require a disproportionately high effort. To narrow the spectrum, we propose a general approach on how to automatically expose systematic differences between human and machine translations to human experts. Inspired by adversarial settings, we train a neural text classifier to distinguish human from machine translations. A classifier that performs and generalizes well after training should recognize systematic differences between the two classes, which we uncover with neural explainability methods. Our proof-of-concept implementation, DiaMaT, is open source. Applied to a dataset translated by a state-of-the-art neural Transformer model, DiaMaT achieves a classification accuracy of 75% and exposes meaningful differences between humans and the Transformer, amidst the current discussion about human parity.<br/>