Andrew Gelman on Social Science, Small Samples, and the Garden of the Forking Paths
What this covers
Andrew Gelman and Russ Roberts examine why statistical findings in psychology and economics routinely fail to replicate, despite appearing rigorous when first published. Gelman argues the root cause is not fraud but a structural problem: researchers face many choices about how to analyze a dataset, and only one successful result needs to emerge to get published. This analytic flexibility—the garden of forking paths—inflates false positives even among honest researchers who do not realize they are engaged in p-hacking. The conversation traces how this happens in practice, why statistical significance is routinely misdefined, and what the replication failures in high-profile studies (priming, ESP, and medical reversals) reveal about the incentive structure of science rather than intentional misconduct.
A central tension emerges: small, noisy samples create a cruel arithmetic. Because only large effects can cross the threshold for statistical significance, published estimates from small samples are systematically inflated through selection bias. Gelman illustrates this with cases ranging from a Cornell study claiming ESP to Daniel Kahneman's acknowledgment that priming effects in his own influential book were likely unreliable. The conversation moves beyond diagnosis to competing visions of reform. One path tightens control: pre-registration and specification before analysis. The other—which Gelman emphasizes—builds statistical methods that openly integrate theory with data, drawing on Bayesian Integration of Theory and Data and examples from baseball analytics and pharmacology where domain knowledge anchors inference. The exchange closes by questioning whether statistical training too often becomes a way to dress ideology in Greek letters rather than a genuine path to learning.
Gelman argues that statistical significance in small, noisy samples systematically overstates effects and fails to replicate—not because of fraud but because the 'garden of forking paths' (analytic flexibility) inflates false positives—so social science should abandon significance-chasing in favor of integrating theory, observational data, and explicit prior information.
- Statistical significance answers the wrong question and is routinely misdefined even by experts
- Small noisy samples force any significant estimate to be large, producing biased overestimates via selection
- Replication failures (priming, ESP) reveal that uncontrolled analyses with many forking paths generate spurious wins
Peer review and statistical significance thresholds do not reliably validate scientific findings.
- The 1-in-20 (5%) threshold for statistical significance is an arbitrary cut-point that has emerged as a norm in published academic research rather than a principled boundary.
“That's an arbitrary cut-point that has somehow emerged as a norm in published academic research. Correct?”
- Daniel Kahneman conceded that priming results in his book Thinking Fast and Slow were probably unreliable, and revealingly admitted he had accepted them because they survived peer review—exposing the flawed assumption that passing peer review is sufficient to establish scientific validity.
“He said, 'Well, I just assumed that, since they had survived the peer-review process, I have to accept them as scientific.' And that was the most stunning thing”
Significant published results frequently fail to replicate or hold at scale in consequential real-world applications.
- As documented in Adam Cifu's Medical Reversal, many multivariate analyses showing statistical significance for new techniques or devices fail to hold up when subjected to randomized controlled trials—a particularly consequential version of the replication failure because it involves life, death, and large sums of money.
“many, many, many multivariate analyses and studies which show statistical significance of some new technique, or new device--when then put into a randomized control trial, do not hold up. It's another version of this kind of failure to replicate.”
- Enthusiasm for deworming charities was driven by a field experiment showing large benefits across many dimensions, but when applied on a very large scale it did not work as well; as with priming, the underlying intervention may be net beneficial, with the magnitude and individual variation being the real points of contention.
“there was a huge encouragement to give to charities that deworm poor children in Africa because a field experiment that found that they did much better in all kinds of dimensions. And when they did it on a very large scale, it didn't work so well.”
Statistical training often masks ideology and intuition rather than illuminating truth.
- There is a danger that statistical training functions less as a path to truth than as a cudgel to beat intellectually inferior opponents, with conclusions ultimately driven by ideology and gut instinct dressed up in Greek letters and appendices; intellectual honesty might favor admitting that a position is mostly gut plus a few facts rather than pretending it is purely data-driven.
“isn't it possible that the study of statistics, the way it's taught... is it just giving you a cudgel, a club, a stick, with which to beat your intellectually inferior opponents? And all it really comes down to is ideology and gut instinct?”