KronA: Parameter Efficient Tuning with Kronecker Adapter

• DOI: 10.48550/arxiv.2212.10650
• Publication Date: 2022-01-01
• Authors: Ali Edalati; Marzieh S. Tahaei; Ivan Kobyzev; Vahid Partovi Nia; James J. Clark; Mehdi Rezagholizadeh
• Keywords: Adapter (computing); Kronecker product; Subnetwork; Representation

Fine-tuning a Pre-trained Language Model (PLM) on specific downstream task has been well-known paradigm in Natural Processing. However, with the ever-growing size of PLMs, training entire model several tasks becomes very expensive and resource-hungry. Recently, different Parameter Efficient Tuning (PET) techniques are proposed to improve efficiency fine-tuning PLMs. One popular category PET methods is low-rank adaptation which insert learnable truncated SVD modules into original either sequentially or parallel. decomposition suffers from limited representation power. In this work, we address problem using Kronecker product instead representation. We introduce KronA, product-based adapter module for efficient Transformer-based apply T5 GLUE benchmark show that incorporating Kronecker-based can outperform state-of-the-art methods.