Facilitating Long Context Understanding via Supervised Chain-of-Thought Reasoning

• DOI: 10.48550/arxiv.2502.13127
• Publication Date: 2025-02-18
• Authors: Jingyang Lin; Andy Kin On Wong; Xia Tian; Shenghua He; Hui Wei; Mei Han; Jiebo Luo
• Keywords: Chain (unit)

Recent advances in Large Language Models (LLMs) have enabled them to process increasingly longer sequences, ranging from 2K 2M tokens and even beyond. However, simply extending the input sequence length does not necessarily lead effective long-context understanding. In this study, we integrate Chain-of-Thought (CoT) reasoning into LLMs a supervised manner facilitate To achieve this, introduce LongFinanceQA, synthetic dataset financial domain designed improve reasoning. Unlike existing data, LongFinanceQA includes intermediate CoT before final conclusion, which encourages perform explicit reasoning, improving accuracy interpretability generate propose Property-driven Agentic Inference (PAI), an agentic framework that simulates human-like steps, including property extraction, retrieval, summarization. We evaluate PAI's capabilities by assessing GPT-4o-mini w/ PAI on Loong benchmark, outperforming standard 20.0%. Furthermore, fine-tune LLaMA-3.1-8B-Instruct achieving 24.6% gain Loong's subset.