Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process

• DOI: 10.1371/journal.pone.0274429
• Publication Date: 2023-01-26T18:29:49Z
• Authors: Hannah Fraser; Martin Bush; Bonnie C. Wintle; Fallon Mody; Eden T. Smith; Anca M. Hanea; Elliot Gould; Victoria Hemming; Daniel G. Hamilton; Libby Rumpff; David P. Wilkinson; Ross Pearson; Felix Singleton T...; Raquel Ashton; Aaron Willcox; Charles T. Gray; Andrew Head; Melissa Ross; Rebecca Groenewegen; Alexandru Marcoci; Ans Vercammen; Timothy H. Parker; Rink Hoekstra; Shinichi Nakagawa; David R. Mandel; Don van Ravenzwaaij; Marissa McBride; Richard O. Sinnott; Peter Vesk; Mark Burgman; Fiona Fidler
• Keywords: Expert elicitation; Replication; Delphi Method; Crowdsourcing; Trustworthiness

As replications of individual studies are resource intensive, techniques for predicting the replicability required. We introduce repliCATS (Collaborative Assessments Trustworthy Science) process, a new method eliciting expert predictions about research. This process is structured elicitation approach based on modified Delphi technique applied to evaluation research claims in social and behavioural sciences. The utility processes predict their capacity test scientific without costs full replication. Experimental data supports validity this with validation study producing classification accuracy 84% an Area Under Curve 0.94, meeting or exceeding other used replicability. provides benefits. It highly scalable, able be deployed both rapid assessment small numbers claims, high volumes over extended period through online platform, having been assess 3000 18 month period. available implemented range ways we describe one such implementation. An important advantage that it collects qualitative has potential provide insight understanding limits generalizability claims. primary limitation its reliance human-derived consequent terms participant fatigue although careful design can minimise these costs. applications alternative peer review allocation effort replication studies.