Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Facts

• DOI: 10.48550/arxiv.2405.11613
• Publication Date: 2024-05-19
• Authors: Baolong Bi; Shenghua Liu; Lingrui Mei; Yiwei Wang; Pengliang Ji; Xueqi Cheng
• Keywords: FOS: Computer and information sciences; Computer Science - Computation and Language; Computation and Language (cs.CL)

The knowledge within large language models (LLMs) may become outdated quickly. While in-context editing (ICE) is currently the most effective method for (KE), it constrained by black-box modeling of LLMs and thus lacks interpretability. Our work aims to elucidate superior performance ICE on KE analyzing impacts new token-wise distributions. We observe that despite a significant boost in logits knowledge, still hindered stubborn knowledge. Stubborn refers as facts have gained excessive confidence during pretraining, making hard edit effectively. To address this issue further enhance ICE, we propose novel approach termed $\textbf{De}$coding $\textbf{C}$ontrasting $\textbf{K}$nowledge (DeCK). DeCK derives distribution next token contrasting obtained from newly edited guided with those unedited parametric experiments consistently demonstrate enhances facts. For instance, improves LLaMA3-8B-instruct MQuAKE up 219%, demonstrating its capability strengthen paves way develop both accountable methods LLMs. (The source code available at: https://deck-llm.meirtz.com)