ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning

• DOI: 10.48550/arxiv.2111.10952
• Publication Date: 2021-01-01
• Authors: Aribandi, Vamsi; Tay, Yi; Schuster, Tal; Rao, Jinfeng; Zheng, Huaixiu St...; Mehta, Sanket Vai...; Zhuang, Honglei; Tran, Vinh Q.; Bahri, Dara; Ni, Jianmo; Gupta, Jai; Hui, Kai; Ruder, Sebastian; Metzler, Donald
• Keywords: FOS: Computer and information sciences; Computer Science - Machine Learning; Computer Science - Computation and Language; 01 natural sciences; Computation and Language (cs.CL); 0105 earth and related environmental sciences; Machine Learning (cs.LG)

ICLR 2022; see https://youtu.be/FbRcbM4T-50 for a video overview of the paper<br/>Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP), few works have systematically studied the effect of scaling up the number of tasks during pre-training. Towards this goal, this paper introduces ExMix (Extreme Mixture): a massive collection of 107 supervised NLP tasks across diverse domains and task-families. Using ExMix, we study the effect of multi-task pre-training at the largest scale to date, and analyze co-training transfer amongst common families of tasks. Through this analysis, we show that manually curating an ideal set of tasks for multi-task pre-training is not straightforward, and that multi-task scaling can vastly improve models on its own. Finally, we propose ExT5: a model pre-trained using a multi-task objective of self-supervised span denoising and supervised ExMix. Via extensive experiments, we show that ExT5 outperforms strong T5 baselines on SuperGLUE, GEM, Rainbow, Closed-Book QA tasks, and several tasks outside of ExMix. ExT5 also significantly improves sample efficiency while pre-training.<br/>