Statistical significance and its critics: practicing damaging science, or damaging scientific practice?

• DOI: 10.1007/s11229-022-03692-0
• Publication Date: 2022-05-12T08:49:54Z
• Authors: Deborah G. Mayo; David Hand
• Keywords: Philosophy; 03 medical and health sciences; 2203 Philosophy; 0302 clinical medicine; 330; 0801 Artificial Intelligence and Image Processing; 2202 History and Philosophy of Specific Fields; Original Research

AbstractWhile the common procedure of statistical significance testing and its accompanying concept of p-values have long been surrounded by controversy, renewed concern has been triggered by the replication crisis in science. Many blame statistical significance tests themselves, and some regard them as sufficiently damaging to scientific practice as to warrant being abandoned. We take a contrary position, arguing that the central criticisms arise from misunderstanding and misusing the statistical tools, and that in fact the purported remedies themselves risk damaging science. We argue that banning the use of p-value thresholds in interpreting data does not diminish but rather exacerbates data-dredging and biasing selection effects. If an account cannot specify outcomes that will not be allowed to count as evidence for a claim—if all thresholds are abandoned—then there is no test of that claim. The contributions of this paper are: To explain the rival statistical philosophies underlying the ongoing controversy; To elucidate and reinterpret statistical significance tests, and explain how this reinterpretation ameliorates common misuses and misinterpretations; To argue why recent recommendations to replace, abandon, or retire statistical significance undermine a central function of statistics in science: to test whether observed patterns in the data are genuine or due to background variability.