Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data

• DOI: 10.18653/v1/2022.acl-long.226
• Publication Date: 2022-06-03T01:34:53Z
• Authors: Shuohang Wang; Yichong Xu; Yuwei Fang; Yang Liu; Siqi Sun; Ruochen Xu; Chenguang Zhu; Michael Zeng
• Keywords: FOS: Computer and information sciences; Computer Science - Computation and Language; Artificial Intelligence (cs.AI); Computer Science - Artificial Intelligence; Computation and Language (cs.CL); 01 natural sciences; Information Retrieval (cs.IR); Computer Science - Information Retrieval; 0105 earth and related environmental sciences

Retrieval-based methods have been shown to be effective in NLP tasks via introducing external knowledge. However, the indexing and retrieving of large-scale corpora bring considerable computational cost. Surprisingly, we found that REtrieving from the traINing datA (REINA) only can lead to significant gains on multiple NLG and NLU tasks. We retrieve the labeled training instances most similar to the input text and then concatenate them with the input to feed into the model to generate the output. Experimental results show that this simple method can achieve significantly better performance on a variety of NLU and NLG tasks, including summarization, machine translation, language modeling, and question answering tasks. For instance, our proposed method achieved state-of-the-art results on XSum, BigPatent, and CommonsenseQA. Our code is released, https://github.com/microsoft/REINA .<br/>Accept to ACL 2022 main conference<br/>