researcher degrees of freedom
The many discretionary choices a researcher makes during data collection and analysis (exclusions, covariates, transformations) that can be exploited, often unconsciously, to produce favorable results.
As defined in Brian Nosek on Truth, Science, and Academic Incentives
Where it comes up · 3
- Ed Yong on Science, Replication, and Journalism4 claims
- Brian Nosek on Truth, Science, and Academic Incentives2 claims
- Andrew Gelman on Social Science, Small Samples, and the Garden of the Forking Paths1 claim
Most important claims · 7 of 7Search all claims for “researcher degrees of freedom” →
- Because there are many different analyses a researcher could run on any dataset and only one needs to be statistically significant to get published, the true chance of finding a 'significant' result under the null is far higher than 5%—like a lottery where you keep buying tickets until you win—even without any conscious cheating.Andrew Gelman on Social Science, Small Samples, and the Garden of the Forking Paths
- Even without conscious fraud, researchers make many analysis decisions (which participants to exclude, which covariates to include) and are subtly influenced to choose those that make results look better; these unconstrained choices leverage chance to produce favorable results, which is why a constrained replication using the same fixed choices often fails to reproduce the original effect.Brian Nosek on Truth, Science, and Academic Incentives
- For strong confirmatory tests, writing up and registering the full hypothesis and analysis plan before accessing the data dramatically reduces researcher degrees of freedom; this is appropriate only for confirmatory work, not for genuinely exploratory research where no strong prediction exists.Brian Nosek on Truth, Science, and Academic Incentives
- Researchers commonly make on-the-fly analytical decisions—checking significance levels during data collection and stopping when significant, not pre-specifying sample size or statistical tests, and discarding outliers—that virtually guarantee positive results; Joe Simmons demonstrated these 'researcher degrees of freedom' could 'show' that listening to the Beatles' 'When I'm Sixty-Four' physically made people younger.Ed Yong on Science, Replication, and Journalism
- The individual problems in science—over-reliance on conceptual replication, researcher degrees of freedom, publication bias, and the difficulty of interpreting replication failures—are each problematic on their own, but synergistically they work together to enhance each other's failings rather than adding up linearly.Ed Yong on Science, Replication, and Journalism
- A survey by Leslie John found that these questionable research practices are not merely possible but actually widespread, with roughly 40-50% of surveyed psychologists admitting to engaging in such data-flexibility tweaks.Ed Yong on Science, Replication, and Journalism
- Psychology relies heavily on 'conceptual replication'—taking many different experiments that test the same underlying idea and treating them as supporting each other—but this only validates a concept if each individual study is itself solid; if the underlying studies were produced through researcher degrees of freedom or selected by publication bias, then conceptual replication becomes the epitome of confirmation bias rather than genuine corroboration.Ed Yong on Science, Replication, and Journalism