replication
Re-doing a study to check its results, either by re-running the original data (narrow) or by re-testing the same question with new data and methods (broad).
As defined in Betsey Stevenson and Justin Wolfers on Happiness, Growth, and the Reinhart-Rogoff Controversy
Other definitions
In options trading and dynamic hedging, the practice of replicating the payoff of one financial instrument using a combination of other instruments; if prices are perfect, replicated instruments should have equal price
A conversation between Nassim Nicholas Taleb and Stephen Wolfram at the Wolfram Summer School 2021
Where it comes up · 16
- Betsey Stevenson and Justin Wolfers on Happiness, Growth, and the Reinhart-Rogoff Controversy5 claims
- Is Most Published Research Wrong?4 claims
- Brian Nosek on the Reproducibility Project4 claims
- Adam Mastroianni on Peer Review and the Academic Kitchen4 claims
- Brian Nosek on Truth, Science, and Academic Incentives3 claims
- Ed Yong on Science, Replication, and Journalism2 claims
- Campbell Harvey on Randomness, Skill, and Investment Strategies2 claims
- Noah Smith on Whether Economics is a Science2 claims
and 8 more
Most important claims · 10 of 35Search all claims for “replication” →
- It is not a coincidence that fraud is surfacing as fields begin validating each other's work; in a culture where no one checks anyone, there is little incentive to refrain from fraud because the chance of being caught is low. Building a culture where work is routinely checked supplies an incentive beyond conscience to do research correctly, catching both purposeful and accidental errors.Betsey Stevenson and Justin Wolfers on Happiness, Growth, and the Reinhart-Rogoff Controversy
- A failed replication does not unambiguously mean the original result is false; it could also reflect subtle but important and unreported nuances in procedure, materials, timing, or expertise, so systematically building cumulative science requires identifying predictors of reproducibility separate from the original being false, including engaging the original researchers before data collection.Brian Nosek on Truth, Science, and Academic Incentives
- Changing behavior permanently is very different from changing it during an active intervention, and almost nothing scales—both the durability of behavior change after a program ends and the failure of interventions to scale are enormous, recurring problems in the field.Susan Mayer on What Money Can't Buy
- Psychology's replication crisis stems from running many studies and publishing only the ~1-in-20 that hits 5% significance while discarding the rest, an under-reporting problem. Economics may be worse, because instead of costly lab experiments, economists can cheaply download data and run regressions, discarding non-results — making it even easier to sift variables until something is significant. The remedy is testing ideas across many datasets and treating replication as re-testing an idea in a new way, not just re-running someone's data.Betsey Stevenson and Justin Wolfers on Happiness, Growth, and the Reinhart-Rogoff Controversy
- A prestigious PNAS paper claiming people are more honest when they sign an attestation at the top of a form rather than the bottom was exposed a decade later as fabricated: a failed replication first published the original raw data, which showed telltale signs of fabrication such as half the data in a different font and an implausibly uniform distribution of reported mileage.Adam Mastroianni on Peer Review and the Academic Kitchen
- Science—including physics and chemistry—progresses not through single 'thunderbolt' studies that decisively change thinking but through the accumulated weight of many studies using slightly different data, assumptions, and methods that produce consistent results; even establishing the charge of the electron took many iterative experiments, so the high rate of non-replicating studies in economics and psychology is normal and expected.Noah Smith on Whether Economics is a Science
- The word 'replication' is widely misunderstood; the broader and more important sense is taking the same question and answering it a different way with different data to see whether results are robust or highly sensitive to the author's chosen data, specifications, and initial conditions — not merely re-running the original author's exact dataset.Betsey Stevenson and Justin Wolfers on Happiness, Growth, and the Reinhart-Rogoff Controversy
- No single solution fixes research bias; over a dozen complementary families of solutions (replication culture, pre-registration, data sharing, better statistical methods, stronger thresholds, open peer review, transparency) reinforce each other, but they only take hold when multiple stakeholders—journals, institutions, funders, scientists—simultaneously adopt them, as clinical-trial registration only happened once major journals refused to publish unregistered trials.John Ioannidis on Statistical Significance, Economics, and Replication
- For a scientist, the successful publication strategy is to avoid replication studies (since journals rarely publish them and results are likely not significant) and instead test novel and unexpected hypotheses while p-hacking to achieve statistical significance.Is Most Published Research Wrong?
- Replication studies should theoretically allow science to self-correct, but in practice they are rarely published by journals, so if a researcher attempts a replication and fails to find significance, they cannot publish the negative result and may be accused of poor methodology rather than the original finding being false.Is Most Published Research Wrong?