John Ioannidis on Statistical Significance, Economics, and Replication
What this covers
John Ioannidis of Stanford joins Russ Roberts to examine why published research findings—particularly in economics but across many fields—frequently misrepresent reality. The conversation centers on Ioannidis's recent analysis of 6,700 economics studies, which reveals that the field operates with a median statistical power of only 18%, meaning most research is too underpowered to reliably detect true effects. When combined with researchers' tendency to pursue statistically significant results through multiple analytic paths, this environment produces false positives and dramatic exaggeration of effect sizes—often by two- to fourfold. The discussion moves through the mechanism: how low power creates a cruel bind where any signal that does reach significance must be either entirely spurious or grossly inflated, a pattern Ioannidis traces across economics, neuroscience, and medicine.
The conversation ranges across several interconnected problems and proposed remedies. They examine the Garden of Forking Paths—the unconscious multitude of analytic choices researchers make that cannot be easily corrected for, unlike in genetics where variants are counted. They discuss Winner's Curse, the phenomenon where underpowered studies necessarily detect only the largest and most inflated effects. Roberts and Ioannidis consider partial solutions: lowering p-value thresholds, pre-registration of analyses, and meta-analysis, each with inherent limitations. Ioannidis argues that no single fix works; instead, multiple stakeholders—journals, institutions, funders—must simultaneously adopt complementary reforms around transparency and replication culture. The discussion also touches on when observational studies are hopeless (single nutrients and disease, where noise dwarfs signal) versus when large effects make them reliable, the inverse problem of big data (where everything becomes significant), and why acknowledging scientific uncertainty matters more than pretending certainty.
Ioannidis argues that most empirical research—including economics—is dominated by underpowered studies and bias, so statistically significant findings are frequently false positives or grossly exaggerated, and the field needs better designs, stricter thresholds, and a culture of replication rather than faith in single significant results.
- Economics meta-analyses show median statistical power around 18%, with effect sizes commonly exaggerated 2- to 4-fold
- Low power combined with significance-chasing bias inflates both false positives and effect magnitudes (Winner's Curse)
- Fixes like lowering the p-value threshold, pre-registration, and replication are partial and must be adopted by multiple stakeholders to work
Small underpowered studies inevitably generate false positives and grossly exaggerated effect sizes.
- In an underpowered environment with very small samples, a true signal can only be detected if it happens to present itself at a much bigger magnitude than reality—because if it appeared at its true size it wouldn't be significant—so any detected result is either a false positive or an exaggerated effect, a phenomenon called the Winner's Curse.
“this is what we call the Winner's Curse... if you find it, you will find it in a way that will present itself in a much bigger magnitude compared to what it really is. Because, if it presents the way that it really is, it will not be significant.”
- Science depends on cumulative knowledge—the basic premise that we look at the totality of evidence, which tells us more than any single study; even a study's deficiencies are informative because they map a universe of evidence.
“science depends on cumulative knowledge. And that's a very basic premise for science--that we are looking at the totality of evidence. And the totality of evidence is going to tell us something more compared to any single study.”
- Operating in an underpowered environment creates two problems simultaneously: a very high chance of false negatives (missing real signals), and a high risk that any statistically significant result is a false positive or a grossly inflated effect size—the smaller the power, the greater the average inflation of detected effects.
“With low power: lots of false negatives. Second problem: lots of false positives and gross exaggeration of the effect sizes.”
- Many of the biases that operate in research share a common denominator: people are trying to find something rather than trying not to find something, leading them to change analyses, add observations, or keep trying until they get a statistically significant p-value, which—when added to low power—causes false-positive rates and effect exaggeration to skyrocket.
“many of the biases that operate, have their common denominator that people are trying to find something rather than trying not to find something”
- Whether a significant small-sample result is a complete false positive or merely an exaggerated true effect depends on the field's prior probability of real effects: in fields with strong theory and prior evidence you tend to get inflated-but-real effects, whereas in agnostic, high-noise fields searching in the dark, almost any significant result is a false positive.
“there is a continuum of different fields and different priors of how many out of 100 or 1000 or 10,000 hypotheses that we are testing are likely to be hiding something that is genuinely not known”
Science progresses through humility, replication, and honest uncertainty rather than false perfection and dogma.
- Across 159 economics meta-analysis topics covering over 6700 empirical studies and ~64,000 parameter estimates, the economics literature shows the same footprint as neuroscience: mostly small, underpowered studies with similar patterns of bias, with a median statistical power of about 18%.
“the footprint of the economics literature that we analyze was pretty similar to the footprint of neuroscience literature... They both have the same pattern of using mostly small studies and underpowered studies. And having pretty similar patterns of bias. So, we estimated a median statistical power of about, um, 18%”
- In most circumstances using statistical significance with p-values is not the best way to approach scientific questions; in Ioannidis's biomedical fields, null hypothesis significance testing is the appropriate approach only about 20% of the time, with the other 80% better served by other tools.
“about 20% of the time, null hypothesis significance testing would be the way to go, indeed. The other 80%, not at all. Or, very second or third type of choice.”
- There is hardly ever a definitive answer in science; we are just trying to approximate closer to the truth, whatever the truth is, and nothing is 100% definitive.
“which I think is untenable because there's hardly ever a definitive answer; we're just trying to approximate closer to the truth, whatever the truth is”
- A single fraudulent Lancet paper claiming MMR vaccines cause autism led hundreds of millions of people to refuse vaccinating their children, illustrating how a wrong result combined with an over-confident 'science is perfect' message can produce catastrophic public consequences.
“Lancet publishing a paper that MMR [Measles, Mumps, Rubella] vaccines cause autism. And then you have hundreds of millions of people who don't want to vaccinate their children. And, we're heading back to the Middle Ages.”
- Acknowledging that science gets things wrong is not anti-science; science does not work with dogma or absolute truth but with healthy skepticism, replication, and bias reduction, and pretending science is perfect makes it an easy target when promises fail—so accurately conveying uncertainty (e.g. 'this is 60% likely true') is essential to maintaining trust.
“science is not working with dogma. It is not working with absolute truth. It's working with some healthy skepticism... Unless we are accurate about our level of uncertainty, I think we will run into trouble.”
- Because there is endless potential for mistakes, unknown biases, and errors in one's own work, some humility is indispensable to good science; the interesting and nice thing about science is that there is no end to detecting and fixing mistakes, so claiming to have done perfect research is presumptuous.
“some humility is indispensible... saying that I have detected the final mistake and now I have been doing perfect research, that's very presumptuous.”
- For the minimum-wage-on-employment question, the median statistical power across studies is about 8.5%, meaning half of the studies had an 8.5% or lower chance of detecting the effect even if the effect suggested by the largest studies is real.
“for that particular question, the median power if I recall that we estimated was something like 8 or 9%”
- In the economics topics analyzed, it is plausible that the large majority of reported effect sizes are substantially exaggerated, with a 2-fold exaggeration very common and about a third of cases inflated by 4-fold.
“It's very common to see an exaggeration of 2-fold. And about 30% or a third would be inflated by 4-fold.”
Observational designs cannot resolve questions where noise vastly exceeds signal; randomized trials are required.
- For questions where the noise is many orders of magnitude larger than the signal—such as single nutrients linked to disease outcomes—observational designs should be abandoned because they cannot recover such tiny effects; randomized or experimental trials minimizing confounding are needed, and continuing observational studies is largely a waste that may bias our thinking rather than provide preliminary insight.
“We should just decide that we are talking about very small effects. The noise is many orders of magnitude more than the signal... So, why are we keep doing this? We should just pause, and abandon this type of design for this type of question.”
- We never needed a randomized trial to prove smoking causes cancer because smoking increases cancer risk roughly 20-fold, whereas many nutrients show effects of only about 1.001-fold; large effect sizes make observational designs reliable while tiny ones do not.
“we never got a randomized trial to prove that smoking causes cancer. But, smoking increases the risk of cancer 20-fold, as opposed by 1.001-fold that many of these nutrients do.”
No single reform fixes bias; journals, institutions, funders, and scientists must simultaneously adopt complementary solutions.
- No single solution fixes research bias; over a dozen complementary families of solutions (replication culture, pre-registration, data sharing, better statistical methods, stronger thresholds, open peer review, transparency) reinforce each other, but they only take hold when multiple stakeholders—journals, institutions, funders, scientists—simultaneously adopt them, as clinical-trial registration only happened once major journals refused to publish unregistered trials.
“registration for clinical trials had been out there as a possibility for 30 years; but it was not really happening until all the major medical journals said, 'I'm not going to publish your trial unless you have pre-registered it.'”
- Pre-registration is a step in the right direction that has helped in domains like clinical trials, but its protection cascades down: roughly 50% of trials are registered, of those ~50% properly, of those ~50% report outcomes, and of those maybe 50% are otherwise well done, so the fully protected fraction shrinks dramatically.
“About 50% of trials are registered; and of those, about 50% are properly registered; and of those that are properly registered, about 50% report their outcomes; and of those, maybe 50% are well done”
- Lowering the conventional significance threshold from 0.05 to 0.005 is a temporizing measure—not a perfect fix—that would eliminate roughly 30% of false positives currently falling in the 0.05–0.005 borderland; it cannot stop researchers from p-hacking toward the new threshold, and when they do succeed the resulting bias and inflation will be worse, but there would be fewer such cases.
“that extra zero is like placing a dam to avoid death by significance... about, on average, 30% of those false positives would no longer be false positives, because they would be in that borderland between 0.05 and 0.005.”
Research bias stems from researchers seeking findings rather than testing them, compounded by untracked analytic choices.
- In many analyses researchers cannot even count their multiplicity burden because they travel down a garden of forking paths—many decision nodes with many options, some choices subconscious—so unlike genetics where variants can be counted and corrected, there is no direct way to correct for the true number of analytic choices made.
“we are going down that garden of forking paths, and we lose count--down the path of how many nodes did we need and how many options were there in each node? And how many choices did we make? And, many of these choices could be even subconscious.”
- Research obtained through a garden of many analytic paths and countless analyses should be honestly acknowledged as entirely exploratory data-dredging rather than dressed up as pre-registered; that is fine provided everyone knows what it is, and the specific emergent recipe can then be pre-registered and tested on a different dataset to see if it replicates.
“What should be conveyed about this research is that it was entirely exploratory, extreme data-dredging at its best; and that's fine, provided that we know that this is what it was.”
Big data produces nominal significance everywhere; meta-analysis reveals bias footprints but not definitive answers.
- Big data studies have the opposite problem of old small studies: they are overpowered, so nominal statistical significance means almost nothing because virtually any analysis will be significant one way or another, leaving significance with little ability to discriminate real findings from flukes.
“we have more and more big data studies, which are over-powered, and where, again, just testing with the typical statistical tools that we have, nominal significance means close to nothing. It's likely that any analysis will be statistically significant one way or another.”
- Looking at big data shows patterns, which is not the same as seeing truth, causal effects, or answers to important questions; these patterns are hard to interpret and depend heavily on theory, so big data should be treated with caution and there are few concrete examples where reliable policy can be built on it—while it is easy to mislead policy with big data.
“looking at big data, you see the patterns in the big data. This is not the same as saying that you see the truth or that you see causal effects or that you see the answer to important questions. You see patterns.”
- Meta-analysis is not automatically more reliable than a single study; if every underlying study is flawed, the meta-analysis yields a flawed result, but it still has value because it reveals the universe of studies and lets you see comparative patterns and footprints of bias rather than promising a definitive answer.
“meta-analysis is not going to fix the literature that is flawed. If every single study is flawed, you will get a flawed result from the meta-analysis. But, you can still get a sense of what is the impact of these flaws”
- Meta-analysis has been under-used in economics: only about 159 topics could be identified, whereas medicine has about 100,000 meta-analyses—roughly three orders of magnitude more—so there is far more accumulated experience about meta-analysis strengths and weaknesses in medicine than in economics.
“if you go to medicine, there is about 100,000 meta-analyses that have published to date... it's a magnitude, or actually 3 orders of magnitude, more, compared to what has been done in economics.”