What this covers

Brian Nosek and Russ Roberts discuss findings from the Reproducibility Project, a systematic attempt to replication 100 studies published in three top psychology journals. The project found that only 36–39% of the original studies' results replicated when independent teams ran the studies again, while the effect size in successful replications was roughly half what the originals reported. Rather than attributing this gap to fraud, Nosek identifies structural incentives as the culprit: journals favor clean, positive findings, and most published work is underpowered, meaning that only inflated, chance-driven effects large enough to cross the statistical significance threshold get published in the first place. The conversation centers on how [reproducibility](concept:Reproducibility as the basis of scientific credibility) is the actual mechanism by which scientific claims gain credibility—not authority or reputation—and why this distinction matters for understanding what "published" and "true" actually mean.

The discussion spans the decline effect, publication bias, and how statistical power and research design shape what makes it into print. Nosek explains why cognitive psychology findings replicate better than social psychology findings, touching on differences between within- vs between-subjects designs. The conversation also covers methodological details—how the project avoided selection bias through transparent protocols and fixed sampling procedures—and addresses how researchers often iterate between data and hypotheses without realizing they've stepped outside the validity boundaries of classical statistics. A recurring theme is the distinction between statistical vs practical significance: a result can be statistically significant yet practically meaningless. The episode closes with broader context: reproducibility problems span all sciences, similar replication projects are now underway in cancer biology and economics, and the issue gained urgency only recently thanks to technology that makes studying publication practices at scale feasible.

Sharpest takeaway

Nosek argues that low reproducibility in published psychology (and science generally) is driven not mainly by fraud but by the incentive to publish clean, significant results combined with underpowered research, and that systematic, transparent replication is essential to grounding scientific credibility.

  • Reproducibility is the core mechanism by which scientific claims gain credibility, not authority or reputation
  • Publication bias plus pervasive low statistical power inflates published effect sizes, explaining the decline effect
  • The Reproducibility Project found only ~36-39% of 100 psychology studies replicated, grounding a previously speculative debate in data

The argument · threads4 threads · 29 claims
0.85

Published results often do not replicate, with replication success tied to original evidence strength rather than who conducted the study.

3 pointscentrality 5/5
  • Across the 100 replications, replication effects were on average half the magnitude of the original effects: 97% of original studies reported statistically significant results but only 36% of replications did, and only 39% of effects were subjectively rated as having replicated.

    “Replication effects were half the magnitude of original effects... Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results”

  • Replication success was better predicted by the strength of the original evidence than by characteristics of the original or replication teams, indicating that the robustness of the underlying finding matters more than who conducted the study.

    “Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.”

  • In development economics, the influential original deworming studies that showed wonderful effects are now under serious reproducibility scrutiny, with re-examination suggesting the effects may be smaller or may not exist at all, paralleling the decline effect seen in psychology.

    “the deworming work is under some seriou[s] issues of reproducibility... Now it doesn't appear--they may not be as large, or they may not exist at all.”

0.65

Researchers invalidate their own statistical tests by iterating between data and hypotheses while continuing to report significance as valid.

1 pointcentrality 5/5
  • When researchers iterate between data and hypotheses—trying different formulations, excluding sample points, adding or removing variables—they leave the framework of classical statistical testing in which significance tests are valid, yet they continue to report and interpret those tests as if they still applied.

    “you've really left... the field of classical statistical testing, and the tests don't apply any more. But we pretend that they do.”

0.61

Statistical significance differs fundamentally from practical importance and does not establish truth, yet journals and media treat publication as proof.

2 pointscentrality 4/5
  • In statistics 'significant' means a result is unlikely to have occurred by chance, which is distinct from the everyday meaning of importance; a statistically significant effect can have an effect size so small that it is practically unimportant, as with the bacon-cancer link.

    “the word 'significant' here, in statistics, is not the same as when it's used in everyday language... it just means that it's unlikely to be true by chance.”

  • The fact that a result is published in a peer-reviewed journal does not establish that it is true; 'published' and 'true' are non-synonyms, a distinction that journalists in particular are incentivized to ignore.

    “'Published and true are non-synonyms.' Which is hard to accept.”

0.34

Failed replications are sometimes dismissed by original authors through ad hoc explanations rather than genuine engagement with the evidence.

2 pointscentrality 2/5
  • Russ jokes that fraudulent results are 'what Photoshop is for' before catching that it is a different field.

    “Well, that's what Photoshop is for. Oh, no, that's a different field.”

  • A high-profile study showing that exposure to words related to the elderly caused people to walk more slowly afterward failed to replicate in an independent attempt, and the original author responded with hostility, attributing the failure to subtle factors like the replicators' tone of voice or pronunciation.

    “they tried to replicate that finding and they couldn't replicate it. And I remember the hostility that the original author gave back to the replicators.”