p-hacking
Manipulating data analysis (e.g., testing many hypotheses or stopping rules) until a statistically significant result appears, inflating false positives.
As defined in ms_ama_november_2025
Where it comes up · 16
- Is Most Published Research Wrong?8 claims
- Andrew Gelman on Social Science, Small Samples, and the Garden of the Forking Paths4 claims
- John Ioannidis on Statistical Significance, Economics, and Replication3 claims
- Betsey Stevenson and Justin Wolfers on Happiness, Growth, and the Reinhart-Rogoff Controversy2 claims
- Tim Harford on the Virtues of Disorder and Messy2 claims
- 150 - The Map Of Misunderstanding2 claims
- John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI1 claim
- Ed Yong on Science, Replication, and Journalism1 claim
and 8 more
Most important claims · 10 of 31Search all claims for “p-hacking” →
- Because there are many different analyses a researcher could run on any dataset and only one needs to be statistically significant to get published, the true chance of finding a 'significant' result under the null is far higher than 5%—like a lottery where you keep buying tickets until you win—even without any conscious cheating.Andrew Gelman on Social Science, Small Samples, and the Garden of the Forking Paths
- Randomized trials and meta-analyses are not infallible truth-tellers; they can be shot through with biases via malleability practices analogous to p-hacking—such as measuring many outcomes and publishing only a subset (one German study found only about 26% of pre-registered outcomes were published) and publication bias of entire trials.Jacob Stegenga on Medical Nihilism
- Researchers should give away their data in the form they used it, show every decision they made, and provide a running tally of all regressions run — not just a vague 'we tried other specifications that didn't change our results' — so reviewers can see how the dish was made and how many times the researcher went cherry-picking; honest disclosure of how many regressions were run creates real credibility.Betsey Stevenson and Justin Wolfers on Happiness, Growth, and the Reinhart-Rogoff Controversy
- P-hacking and the garden of forking paths are not fraud or deliberate manipulation; they are ordinary research practice where the natural inclination to try different specifications when an initial result is uninteresting produces spurious significance, comparable to an unblinded doctor using a biased procedure with good intentions.Andrew Gelman on Social Science, Small Samples, and the Garden of the Forking Paths
- To guard against unreliable single studies, Harford avoids reporting amazingly counterintuitive results that pass significance only marginally (e.g., p=0.04) and instead builds arguments from multiple independent sources—stories, interviews with participants, social psychology, computer science, cognitive psychology, neuroscience—so that even if any single study fails to replicate, convergence from many different angles gives reasonable confidence.Tim Harford on the Virtues of Disorder and Messy
- In many analyses researchers cannot even count their multiplicity burden because they travel down a garden of forking paths—many decision nodes with many options, some choices subconscious—so unlike genetics where variants can be counted and corrected, there is no direct way to correct for the true number of analytic choices made.John Ioannidis on Statistical Significance, Economics, and Replication
- Psychology's replication crisis stems from running many studies and publishing only the ~1-in-20 that hits 5% significance while discarding the rest, an under-reporting problem. Economics may be worse, because instead of costly lab experiments, economists can cheaply download data and run regressions, discarding non-results — making it even easier to sift variables until something is significant. The remedy is testing ideas across many datasets and treating replication as re-testing an idea in a new way, not just re-running someone's data.Betsey Stevenson and Justin Wolfers on Happiness, Growth, and the Reinhart-Rogoff Controversy
- When replications fail, the key difference is usually not the experimental conditions but that the replication is controlled—the analysis is specified in advance—whereas the original study was uncontrolled and permitted p-hacking; even honest researchers like Nosek discovered they had p-hacked without realizing it.Andrew Gelman on Social Science, Small Samples, and the Garden of the Forking Paths
- A Cornell ESP study claiming students could foretell the future was published in a top journal (Journal of Personality and Social Psychology) and treated as impeccable methodology, even though most people including likely the editor disbelieved it—revealing that perceived methodological rigor and peer review can shield deeply flawed, uncontrolled analyses.Andrew Gelman on Social Science, Small Samples, and the Garden of the Forking Paths
- Journals are far more likely to publish results that reach statistical significance (p<0.05), creating incentives for researchers to use analytical methods that produce lower p-values, thus promoting p-hacking as a successful publication strategy.Is Most Published Research Wrong?