K-ON: Stacking Knowledge On the Head Layer of Large Language Model

• DOI: 10.48550/arxiv.2502.06257
• Publication Date: 2025-02-10
• Authors: Lingbing Guo; Yichi Zhang; Zhongpu Bo; Zhuo Chen; Mengshu Sun; Zhiqiang Zhang; Wen Zhang; Hua‐Jun Chen
• Keywords: FOS: Computer and information sciences; Computer Science - Computation and Language; Artificial Intelligence (cs.AI); Computer Science - Artificial Intelligence; Computation and Language (cs.CL)

Recent advancements in large language models (LLMs) have significantly improved various natural processing (NLP) tasks. Typically, LLMs are trained to predict the next token, aligning well with many NLP However, knowledge graph (KG) scenarios, entities fundamental units and identifying an entity requires at least several tokens. This leads a granularity mismatch between KGs languages. To address this issue, we propose K-ON, which integrates KG into LLM by employing multiple head layers for k-step prediction. K-ON can not only generate entity-level results one step, but also enables contrastive loss against entities, is most powerful tool representation learning. Experimental show that outperforms state-of-the-art methods incorporate text even other modalities.