Meta-Analytic Links Between Evidentiary Value and the Practices of Probing Suspicion and Excluding Suspicious Participants

• DOI: 10.31234/osf.io/aupbd_v2
• Publication Date: 2025-04-15T14:58:18Z
• Authors: Wyatt Thomas Brown; David Chester

Suspicion probes in experimental psychology are a widely used methodological tool and are often touted as a gold standard approach to ensure internal validity. However, suspicion probes (A) allow investigators to flexibly exclude participants and analyze their data and (B) remain unvalidated — two factors that can undermine the evidentiary value of the scientific literature that employs these practices. In an initial, broad meta-analytic p-curve of 200 effects from five eminent journals in experimental psychology, we found lower evidentiary value among studies that reported (versus those that did not report) the use of suspicion probes and those that excluded (versus retained) suspicious participants. In a second, more targeted meta-analytic p-curve of 152 effects that all used a similar deceptive aggression paradigm, we found no differences in evidentiary value. In an exploratory fashion, we also sought to examine whether the exclusion of suspicious participants was done in an ethnically and racially equitable fashion, but so few studies (i.e., ~15% of those that excluded suspicious participants) provided adequate demographic information we were unable to conduct these analyses. Though purely correlational, our findings suggest that the use of suspicion probes and the exclusion of suspicious participants offer, at best, no benefit. At worst, such probes and exclusions may undermine the evidentiary value of the scientific literature within experimental psychology.