Efficient Transformers: A Survey

• DOI: 10.48550/arxiv.2009.06732
• Publication Date: 2020-01-01
• Authors: Yi Tay; Mostafa Dehghani; Dara Bahri; Donald Metzler
• Keywords: Performing arts

Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision and reinforcement learning. In the field natural language processing for example, Transformers become an indispensable staple in modern deep learning stack. Recently, dizzying number "X-former" models been proposed - Reformer, Linformer, Performer, Longformer, name few which improve upon original architecture, many make improvements around computational memory efficiency. With aim helping avid researcher navigate this flurry, paper characterizes large thoughtful selection recent efficiency-flavored models, providing organized comprehensive overview existing work multiple domains.