Joint Evaluation of Fairness and Relevance in Recommender Systems with Pareto Frontier

• DOI: 10.1145/3696410.3714589
• Publication Date: 2025-02-17
• Authors: Theresia Veronika...; Tuukka Ruotsalo; Maria Maistro; Christina Lioma
• Keywords: Frontier; Relevance; Pareto optimal

Fairness and relevance are two important aspects of recommender systems (RSs). Typically, they evaluated either (i) separately by individual measures fairness relevance, or (ii) jointly using a single measure that accounts for with respect to relevance. However, approach often does not provide reliable joint estimate the goodness models, as it has different best models: one another Approach is also problematic because these tend be ad-hoc do relate well traditional measures, like NDCG. Motivated this, we present new evaluating in RSs: Distance Pareto Frontier (DPFR). Given some user-item interaction data, compute their frontier pair existing then use distance from achievable Our modular intuitive can computed measures. Experiments 4 RS 3 re-ranking strategies, 6 datasets show metrics have inconsistent associations our Pareto-optimal solution, making DPFR more robust theoretically well-founded assessing code: https://github.com/theresiavr/DPFR-recsys-evaluation