A fuzzy set theory based computational model to represent the quality of inter-rater agreement

• DOI: 10.1007/s11135-013-9888-3
• Publication Date: 2013-07-05T05:57:14Z
• Authors: Enrico Ciavolino; Sergio Salvatore; Antonio Calcagnì
• Keywords: Inter-Rater Reliability; Judgement; Agreement

In this paper we present a method to evaluate the quality of a rater’s judgement, which can integrate and enrich the use of inter-rater agreement as a reliability measure. Our proposal is an integrative one and evaluates the quality of a rater’s performance through an analysis of the profile of that individual rater’s performance. We discuss its rationale on the basis of the interpretation of inter-rater agreement, highlighting some critical issues. For this purpose, we adopt a computational model based on fuzzy set theory, demonstrating its main characteristics with an exemplary case study.