Analyzing Context Utilization of LLMs in Document-Level Translation

• DOI: 10.48550/arxiv.2410.14391
• Publication Date: 2024-10-18
• Authors: Wafaa Mohammed; Vlad Niculae
• Keywords: FOS: Computer and information sciences; Computer Science - Computation and Language; Computation and Language (cs.CL)

Large language models (LLM) are increasingly strong contenders in machine translation. We study document-level translation, where some words cannot be translated without context from outside the sentence. investigate ability of prominent LLMs to utilize by analyzing models' robustness perturbed and randomized document context. find that LLMs' improved document-translation performance is not always reflected pronoun translation performance. highlight need for context-aware finetuning with a focus on relevant parts improve their reliability