The puzzling relationship between multi-lab replications and meta-analyses of the rest of the literature

• DOI: 10.31234/osf.io/pbrdk
• Publication Date: 2020-04-03T17:36:17Z
• Authors: Molly Lewis; Maya B Mathur; Tyler VanderWeele; Michael C. Frank
• Keywords: Value (mathematics); Rest (music)

What is the best way to estimate size of important effects? Should we aggregate across disparate findings using statistical meta-analysis, or instead run large, multi-lab replications (MLR)? A recent paper by Kvarven, Strømland, and Johannesson (2020) compared effect estimates derived from these two different methods for 15 psychological phenomena. The authors report that, same phenomenon, meta-analytic tends be about three times larger than MLR estimate. These results pose an puzzle: relationship between estimates? Kvarven et al. suggest that their undermine value meta-analysis. In contrast, argue both meta-analysis are informative, discrepancy obtained via in fact still unexplained. Informed re-analyses al.’s data other empirical evidence, discuss possible sources this understanding puzzle future meta-scientific research.