A Systematic Study of Performance Disparities in Multilingual Task-Oriented Dialogue Systems

• DOI: 10.18653/v1/2023.emnlp-main.422
• Publication Date: 2023-12-10T21:58:19Z
• Authors: Songbo Hu; Han Zhou; Moy Yuan; Milan Gritta; Guchun Zhang; Ignacio Iacobacci; Anna Korhonen; Ivan Vulić
• Keywords: FOS: Computer and information sciences; Computer Science - Computation and Language; Computation and Language (cs.CL)

Achieving robust language technologies that can perform well across the world’s many languages is a central goal of multilingual NLP. In this work, we take stock and empirically analyse task performance disparities exist between task-oriented dialogue (ToD) systems. We first define new quantitative measures absolute relative equivalence in system performance, capturing within individual languages. Through series controlled experiments, demonstrate depend on number factors: nature ToD at hand, underlying pretrained model, target language, amount annotated data. prove existence adaptation intrinsic biases current systems: e.g., systems trained for Arabic or Turkish using data fully parallel to English still exhibit diminished performance. Beyond providing insights into different languages, our analyses offer practical tips how approach collection development