Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning

• DOI: 10.18653/v1/2023.acl-short.107
• Publication Date: 2023-08-05T00:57:42Z
• Authors: Zhen-Ru Zhang; Chuanqi Tan; Haiyang Xu; Chengyu Wang; Jun Huang; Songfang Huang
• Keywords: Prefix; Fine-tuning

Fine-tuning large pre-trained language models on various downstream tasks with whole parameters is prohibitively expensive. Hence, Parameter-efficient fine-tuning has attracted attention that only optimizes a few task-specific the frozen model. In this work, we focus prefix tuning, which continuous vectors (i.e. pseudo tokens) inserted into Transformer layers. Based observation learned syntax and semantics representation varies lot at different layers, argue adaptive will be further tailored to each layer than fixed one, enabling more effective efficient. Thus, propose Adaptive Prefix Tuning (APT) adjust in terms of both fine-grained token level coarse-grained gate mechanism. Experiments SuperGLUE NER datasets show effectiveness APT. addition, taking as probing, validate efficiency variable prefix.