Measuring the Inconsistency of Large Language Models in Preferential Ranking

• DOI: 10.18653/v1/2024.knowllm-1.14
• Publication Date: 2024-09-20T19:33:08Z
• Authors: Xiutian Zhao; Ke Wang; Wei Peng
• Keywords: FOS: Computer and information sciences; Computer Science - Computation and Language; Computation and Language (cs.CL)

Despite large language models' (LLMs) recent advancements, their bias and hallucination issues persist, ability to offer consistent preferential rankings remains underexplored. This study investigates the capacity of LLMs provide ordinal preferences, a crucial aspect in scenarios with dense decision space or lacking absolute answers. We introduce formalization consistency based on order theory, outlining criteria such as transitivity, asymmetry, reversibility, independence from irrelevant alternatives. Our diagnostic experiments selected state-of-the-art reveal inability meet these criteria, indicating strong positional poor preferences easily swayed by These findings highlight significant inconsistency LLM-generated rankings, underscoring need for further research address limitations.