Towards Context-Robust LLMs: A Gated Representation Fine-tuning Approach

• DOI: 10.48550/arxiv.2502.14100
• Publication Date: 2025-02-19
• Authors: Shenglai Zeng; Pengfei He; Kai Guo; Tianqi Zheng; Hanqing Lu; Yue Xing; Hui Liu
• Keywords: Representation

Large Language Models (LLMs) enhanced with external contexts, such as through retrieval-augmented generation (RAG), often face challenges in handling imperfect evidence. They tend to over-rely on knowledge, making them vulnerable misleading and unhelpful contexts. To address this, we propose the concept of context-robust LLMs, which can effectively balance internal knowledge context, similar human cognitive processes. Specifically, LLMs should rely context only when lacking identify contradictions between disregard achieve this goal, introduce Grft, a lightweight plug-and-play gated representation fine-tuning approach. Grft consists two key components: gating mechanism detect filter problematic inputs, low-rank adapters adjust hidden representations. By training intervention function 0.0004\% model size fewer than 200 examples, adapt towards behaviors.