Diagnostic Evaluation of Information Retrieval Models

• DOI: 10.1145/1961209.1961210
• Publication Date: 2011-05-03T12:48:53Z
• Authors: Hui Fang; Tao Tao; Chengxiang Zhai
• Keywords: Heuristics; Relevance; Implementation; Strengths and weaknesses

Developing effective retrieval models is a long-standing central challenge in information research. In order to develop more models, it necessary understand the deficiencies of current and relative strengths each them. this article, we propose general methodology analytically experimentally diagnose weaknesses function, which provides guidance on how further improve its performance. Our motivated by empirical observation that good performance closely related use various heuristics. We connect function with implementations these heuristics, two strategies check well implements desired The first strategy formalize heuristics as constraints, constraint analysis implementation second define set relevance-preserving perturbations perform diagnostic tests empirically evaluate Experiments show both are identify potential problems functions can be improved after fix problems.