MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

• DOI: 10.48550/arxiv.2405.12130
• Publication Date: 2024-05-20
• Authors: Ting Jiang; Shaohan Huang; Shengyue Luo; Zihan Zhang; Haizhen Huang; Furu Wei; Wei‐Wei Deng; Feng Sun; Qi Zhang; Deqing Wang; Fuzhen Zhuang
• Keywords: Rank (graph theory)

Low-rank adaptation is a popular parameter-efficient fine-tuning method for large language models. In this paper, we analyze the impact of low-rank updating, as implemented in LoRA. Our findings suggest that updating mechanism may limit ability LLMs to effectively learn and memorize new knowledge. Inspired by observation, propose called MoRA, which employs square matrix achieve high-rank while maintaining same number trainable parameters. To it, introduce corresponding non-parameter operators reduce input dimension increase output matrix. Furthermore, these ensure weight can be merged back into LLMs, makes our deployed like We perform comprehensive evaluation across five tasks: instruction tuning, mathematical reasoning, continual pretraining, memory pretraining. outperforms LoRA on memory-intensive tasks achieves comparable performance other tasks.