Retrieval-Augmented Large Language Models (LLMs), which incorporate the non-parametric knowledge from external bases into LLMs, have emerged as a promising approach to enhancing response accuracy in several tasks, such Question-Answering (QA). However, even though there are various approaches dealing with queries of different complexities, they either handle simple unnecessary computational overhead or fail adequately address complex multi-step queries; yet, not all user requests fall only...

Accelerating inference in Large Language Models (LLMs) is critical for real-time interactions, as they have been widely incorporated into real-world services. Speculative decoding, a fully algorithmic solution, has gained attention improving speed by drafting and verifying tokens, thereby generating multiple tokens single forward pass. However, current strategies usually require significant fine-tuning or inconsistent performance across tasks. To address these challenges, we propose...

Dense retrieval models, which aim at retrieving the most relevant document for an input query on a dense representation space, have gained considerable attention their remarkable success. Yet, models require vast amount of labeled training data notable performance, whereas it is often challenging to acquire query-document pairs annotated by humans. To tackle this problem, we propose simple but effective Document Augmentation Retrieval (DAR) framework, augments representations documents with...

Re-rankers, which order retrieved documents with respect to the relevance score on given query, have gained attention for information retrieval (IR) task. Rather than fine-tuning pre-trained language model (PLM), large-scale (LLM) is utilized as a zero-shot re-ranker excellent results. While LLM highly dependent prompts, impact and optimization of prompts are not explored yet. Along highlighting re-ranker, we propose novel discrete prompt method, Constrained Prompt generation (Co-Prompt),...

Re-rankers, which order retrieved documents with respect to the relevance score on given query, have gained attention for information retrieval (IR) task. Rather than fine-tuning pre-trained language model (PLM), large-scale (LLM) is utilized as a zero-shot re-ranker excellent results. While LLM highly dependent prompts, impact and optimization of prompts are not explored yet. Along highlighting re-ranker, we propose novel discrete prompt method, Constrained Prompt generation (Co-Prompt),...

Large language models (LLMs) enable zero-shot approaches in open-domain question answering (ODQA), yet with limited advancements as the reader is compared to retriever. This study aims at feasibility of a that addresses challenges computational cost and need for labeled data. We find LLMs are distracted due irrelevant documents retrieved set overconfidence generated answers when they exploited readers. To tackle these problems, we mitigate impact such via Distraction-aware Answer Selection...

Information retrieval models that aim to search for the documents relevant given query have shown many successes, which been applied diverse tasks. However, provided by user is oftentimes very short, challenges retrievers correctly fetch documents. To tackle this, existing studies proposed expanding with a couple of additional (user-related) features related query. Yet, they may be suboptimal effectively augment query, though there plenty information available it in relational database....

Sukmin Cho, Soyeong Jeong, Wonsuk Yang, Jong Park. Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Architectures. 2022.

The robustness of recent Large Language Models (LLMs) has become increasingly crucial as their applicability expands across various domains and real-world applications. Retrieval-Augmented Generation (RAG) is a promising solution for addressing the limitations LLMs, yet existing studies on RAG often overlook interconnected relationships between components or potential threats prevalent in databases, such minor textual errors. In this work, we investigate two underexplored aspects when...

Recent language models have shown remarkable performance on natural understanding (NLU) tasks. However, they are often sub-optimal when faced with ambiguous samples that can be interpreted in multiple ways, over-confidently predicting a single label without consideration for its correctness. To address this issue, we propose novel self-knowledge distillation method enables to learn distributions more accurately by leveraging knowledge distilled from their lower layers. This approach also...

Recent advancements in Large Language Models (LLMs) have significantly improved their performance across various Natural Processing (NLP) tasks. However, LLMs still struggle with generating non-factual responses due to limitations parametric memory. Retrieval-Augmented Generation (RAG) systems address this issue by incorporating external knowledge a retrieval module. Despite successes, however, current RAG face challenges failures and the limited ability of filter out irrelevant information....

We introduce EXIT, an extractive context compression framework that enhances both the effectiveness and efficiency of retrieval-augmented generation (RAG) in question answering (QA). Current RAG systems often struggle when retrieval models fail to rank most relevant documents, leading inclusion more at expense latency accuracy. While abstractive methods can drastically reduce token counts, their token-by-token process significantly increases end-to-end latency. Conversely, existing but rely...

Dense retrieval models, which aim at retrieving the most relevant document for an input query on a dense representation space, have gained considerable attention their remarkable success. Yet, models require vast amount of labeled training data notable performance, whereas it is often challenging to acquire query-document pairs annotated by humans. To tackle this problem, we propose simple but effective Document Augmentation Retrieval (DAR) framework, augments representations documents with...

Recent instruction-finetuned large language models (LMs) have achieved notable performances in various tasks, such as question-answering (QA). However, despite their ability to memorize a vast amount of general knowledge across diverse they might be suboptimal on specific tasks due limited capacity transfer and adapt target tasks. Moreover, further finetuning LMs with labeled datasets is often infeasible absence, but it also questionable if we can smaller having only unlabeled test data. In...

Large language models (LLMs) enable zero-shot approaches in open-domain question answering (ODQA), yet with limited advancements as the reader is compared to retriever. This study aims at feasibility of a that addresses challenges computational cost and need for labeled data. We find LLMs are distracted due irrelevant documents retrieved set overconfidence generated answers when they exploited readers. To tackle these problems, we mitigate impact such via Distraction-aware Answer Selection...

Recent instruction-finetuned large language models (LMs) have achieved notable performances in various tasks, such as question-answering (QA). However, despite their ability to memorize a vast amount of general knowledge across diverse they might be suboptimal on specific tasks due limited capacity transfer and adapt target tasks. Moreover, further finetuning LMs with labeled datasets is often infeasible absence, but it also questionable if we can smaller having only unlabeled test data. In...