Are Key-Phrases All That Reviewers Care About? A Comprehensive Benchmarking of Reviewer Matchmaking Systems

• DOI: 10.1609/aaai.v39i22.34545
• Publication Date: 2025-04-11T13:26:07Z
• Authors: Sourish Dasgupta; Harsh Sharma; Devansh Patel; Prarthee Desai; Anil K. Roy
• Keywords: Benchmarking

Reviewer Matchmaking (RM) is a pivotal process in academic publishing that aligns manuscripts with appropriate reviewers based on their expertise and prior publications. The demand for an automated RM system has escalated the significant surge submissions over past decade. State-of-the-art (SOTA) models are document-representation-based (DR-RM) match manuscript reviewer's publication using similarity method defined high-dimensional vector space. However, they far from accurate despite large-scale usage. In this paper, we establish conventional evaluation measures unreliable instead emphasize standard correlation adequate. For first time, compare performance of six SOTA DR-RM those fourteen Key-phrase Extraction-based (KPE-RM) - alternate unexplored approach. We observe KPE-RM show comparable results many cases, new best model being PatternRank-RM beating SPECTER2-RM (Pearson: 0.004+, Spearman: 0.006+, Kendall: 0.043+). conclude must be contextualized to task cannot used as plug-n-play.