XPrompt: Exploring the Extreme of Prompt Tuning

• DOI: 10.18653/v1/2022.emnlp-main.758
• Publication Date: 2023-08-04T20:21:02Z
• Authors: Fang Ma; Chen Zhang; Lei Ren; Jingang Wang; Qifan Wang; Wei Wu; Xiaojun Quan; Dawei Song
• Keywords: Granularity; Pruning; Fine-tuning; Bridge (graph theory)

Prompt tuning learns soft prompts to condition the frozen Pre-trained Language Models (PLMs) for performing downstream tasks in a parameter-efficient manner. While prompt has gradually reached performance level of fine-tuning as model scale increases, there is still large gap between and models moderate small scales (typically less than 11B parameters). In this paper, we empirically show that trained tokens can have negative impact on task thus degrade its performance. To bridge gap, propose novel with an eXtremely (XPrompt) under regime lottery tickets hypothesis. Specifically, XPrompt eliminates at different granularity levels through hierarchical structured pruning, yielding more yet competitive Comprehensive experiments are carried out SuperGLUE tasks, results indicate able close smaller scales.