Replication Crisis
An ongoing problem in fields like psychology where many published research findings cannot be reproduced in subsequent studies, casting doubt on their universal validity.
As defined in bigthink_limits_of_rationalism
Other definitions
A documented failure of peer review and scientific integrity in which published research findings cannot be replicated by independent researchers, indicating systematic problems in knowledge production and validation.
Why the world is going crazy—and how to win back our minds | Jamie Wheal | Big ThinkThe finding that many published social science and psychology results fail to reproduce when studies are repeated.
Tyler Cowen & Daniel Gross - Identifying Talent - [Invest Like the Best, EP. 277]
Where it comes up · 39
- 95 | Liam Kofi Bright on Knowledge, Truth, and Science4 claims
- Andrew Gelman on Social Science, Small Samples, and the Garden of the Forking Paths3 claims
- Katy Milkman on How to Change3 claims
- Michele Gelfand2 claims
- Charlan Nemeth on In Defense of Troublemakers2 claims
- 150 - The Map Of Misunderstanding2 claims
- 158 - Understanding Humans In The Wild2 claims
- Why the world is going crazy—and how to win back our minds | Jamie Wheal | Big Think2 claims
and 31 more
Most important claims · 10 of 51Search all claims for “Replication Crisis” →
- When replications fail, the key difference is usually not the experimental conditions but that the replication is controlled—the analysis is specified in advance—whereas the original study was uncontrolled and permitted p-hacking; even honest researchers like Nosek discovered they had p-hacked without realizing it.Andrew Gelman on Social Science, Small Samples, and the Garden of the Forking Paths
- Multiple biases beyond publication bias distort scientific behavior: demographic biases (e.g., gender) in who gets published and where; biases toward results the media or policymakers will pick up on—which can drive outright fraud chasing clout; and scientists' own desire to change the world in directions they find desirable. Some (like the file drawer problem) admit relatively simple statistical or editorial fixes, but gender bias and the subtle pull of wanting to be influential are much harder to correct.95 | Liam Kofi Bright on Knowledge, Truth, and Science
- Peer review is the least of psychology's problems; the field faces the replication crisis (statistics done in ways that inflated significant results), the WEIRD-sample problem (a randomly selected American undergraduate is 4,000 times more likely to be a study subject than someone outside the West), the file-drawer problem, and political and economic biases—but it is improving by running bigger studies and having itself uncovered these problems.Paul Bloom on Psych, Psychology, and the Human Mind
- In the 'Universe of Uncertainty' study, 76 non-communicating research teams given identical data on whether immigration affects support for the social safety net produced scattered results—about half found no effect, a quarter found a statistically significant positive effect, and a quarter found a statistically significant negative effect—revealing that even with controlled data and responsible methodologies, social research yields scattershot conclusions that publication bias then cements into 'settled' policy-shaping wisdom.If Life Is Random, Is It Meaningless? (with Brian Klaas)
- The file drawer problem biases the scientific literature: positive results suggesting a new effect are exciting and get published in high-prestige journals read by many, while null results or replications are hard to publish and often remain only with the scientist who produced them—creating a massive bias in what information reaches the community. This problem was known long before the replication crisis, which has prompted cleaning up house that should have happened earlier.95 | Liam Kofi Bright on Knowledge, Truth, and Science
- Replication is far easier in psychology than in economics or political science because you cannot deliberately rerun social experiments—you can't start wars to study international relations or create recessions to study the economy—and historical events resist generalization because each can be claimed to be uniquely different.Andrew Gelman on Social Science, Small Samples, and the Garden of the Forking Paths
- The natural incentive to hide uninteresting or null results means the studies we observe disproportionately come from the tails of the distribution of possible results, making the published literature less reliable, as seen in the replication crisis in psychology and beyond.Julia Belluz on Epidemiology, Nutrition, and Metabolism
- Because human behavior is too complex and contingent for any effect to be robust across all people and conditions, good social science should identify boundary conditions—asking when an effect holds, goes neutral, or reverses—rather than treating the replication crisis as a binary gotcha about whether an effect is real or fake.158 - Understanding Humans In The Wild
- A single social-psychology study, even with careful statistics, should not be treated as a fact set in stone, because one can choreograph almost any study to yield a different finding by altering subject selection, the task, or how terms are operationally defined; responsible knowledge requires integrating many studies and trusting some more than others based on methodological scrutiny.Charlan Nemeth on In Defense of Troublemakers
- Roughly 300 papers cited the original priming study as 'conceptual replications,' each testing slightly different conditions and finding some pattern, which created an overwhelming illusion of confirmation; this is itself a forking-paths problem because contradictory results can always be rationalized into apparent support.Andrew Gelman on Social Science, Small Samples, and the Garden of the Forking Paths