LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks

• DOI: 10.48550/arxiv.2402.11455
• Publication Date: 2024-02-17
• Authors: Hanqing Wang; Bowen Ping; Shuo Wang; Xu Han; Yun Chen; Zhiyuan Liu; Maosong Sun

LoRA employs lightweight modules to customize large language models (LLMs) for each downstream task or domain, where different learned additional represent diverse skills. Combining existing LoRAs address new tasks can enhance the reusability of LoRAs, particularly beneficial with limited annotated data. Most prior works on combination primarily rely task-level weights involved LoRA, making examples and tokens share same weights. However, in generative tasks, may necessitate skills manage. Taking Chinese math as an example, understanding problem description depend more while calculation part LoRA. To this end, we propose LoRA-Flow, which utilizes dynamic adjust impact LoRAs. The at step are determined by a fusion gate extremely few parameters, be only 200 training examples. Experiments across six demonstrate that our method consistently outperforms baselines This underscores necessity introducing combination.