Alternating Language Modeling for Cross-Lingual Pre-Training

• DOI: 10.1609/aaai.v34i05.6480
• Publication Date: 2020-06-18T08:13:53Z
• Authors: Jian Yang; Shuming Ma; Dongdong Zhang; ShuangZhi Wu; Zhoujun Li; Ming Zhou
• Keywords: Concatenation (mathematics); Code (set theory)

Language model pre-training has achieved success in many natural language processing tasks. Existing methods for cross-lingual adopt Translation Model to predict masked words with the concatenation of source sentence and its target equivalent. In this work, we introduce a novel method, called Alternating Modeling (ALM). It code-switches sentences different languages rather than simple concatenation, hoping capture rich context phrases. More specifically, randomly substitute phrases translations create code-switched sentences. Then, use these data train ALM learn languages. We evaluate our on downstream tasks machine translation classification. Experiments show that can outperform previous three benchmarks.1