LSTM-based Selective Dense Text Retrieval Guided by Sparse Lexical Retrieval

• DOI: 10.48550/arxiv.2502.10639
• Publication Date: 2025-02-14
• Authors: Yingrui Yang; Parker Carlson; Yifan Qiao; Wentai Xie; Shanxiu He; Tao Yang
• Keywords: Text retrieval

This paper studies fast fusion of dense retrieval and sparse lexical retrieval, proposes a cluster-based selective method called CluSD guided by retrieval. takes lightweight approach exploits the overlap results embedding clusters in two-stage selection process with an LSTM model to quickly identify relevant while incurring limited extra memory space overhead. triggers partial performs block disk I/O if needed. evaluates compares it several baselines for searching in-memory on-disk MS MARCO BEIR datasets.