Experimenter evidence unmasking as a confound in (Bayesian) optional stopping

• DOI: 10.31219/osf.io/5f43x
• Publication Date: 2024-04-03T20:54:10Z
• Authors: Renata Sadibolova; Devin B Terhune

Bayesian optional stopping refers to the practice of repeatedly performing a statistical analysis on a dataset as new data are collected until a pre-specified Bayesian evidence criterion is reached. This procedure is becoming increasingly common owing in part to its efficacy in optimizing data collection. Discussions of this procedure to date have been restricted to statistical issues and have omitted consideration of any methodological implications of this procedure. Here we highlight experimenter awareness of the current evidence state during data collection (experimenter evidence unmasking) as a methodological confound in this procedure. Experimenter evidence unmasking has the potential to influence an experimenter to implicitly or explicitly modify their experimental behaviour in ways that can reduce the internal validity of a study by biasing the assessment of an experimental manipulation. We conclude by offering recommendations for circumventing this confound and for the transparent reporting of experimenter masking procedures.