publication bias
The tendency to publish only statistically significant or 'working' results while discarding null findings, distorting the literature.
As defined in Betsey Stevenson and Justin Wolfers on Happiness, Growth, and the Reinhart-Rogoff Controversy
Other definitions
Tendency of journals to publish positive results while negative results go unreported.
Alex Karnal - The Trillion-Dollar Health Revolution - [Invest Like the Best, EP.468]
Where it comes up · 16
- Ed Yong on Science, Replication, and Journalism3 claims
- Is Most Published Research Wrong?2 claims
- Marcia Angell on Big Pharma2 claims
- Brian Nosek on the Reproducibility Project2 claims
- William Byers on the Blind Spot, Science, and Uncertainty 05/16/20111 claim
- Daniel Pink on Drive, Motivation, and Incentives 08/30/20101 claim
- Robin Hanson on Health 5/28/20071 claim
- The Invisible Hierarchies that Rule Our World (with Toby Stuart)1 claim
and 8 more
Most important claims · 10 of 21Search all claims for “publication bias” →
- Randomized trials and meta-analyses are not infallible truth-tellers; they can be shot through with biases via malleability practices analogous to p-hacking—such as measuring many outcomes and publishing only a subset (one German study found only about 26% of pre-registered outcomes were published) and publication bias of entire trials.Jacob Stegenga on Medical Nihilism
- Psychology's replication crisis stems from running many studies and publishing only the ~1-in-20 that hits 5% significance while discarding the rest, an under-reporting problem. Economics may be worse, because instead of costly lab experiments, economists can cheaply download data and run regressions, discarding non-results — making it even easier to sift variables until something is significant. The remedy is testing ideas across many datasets and treating replication as re-testing an idea in a new way, not just re-running someone's data.Betsey Stevenson and Justin Wolfers on Happiness, Growth, and the Reinhart-Rogoff Controversy
- In the 'Universe of Uncertainty' study, 76 non-communicating research teams given identical data on whether immigration affects support for the social safety net produced scattered results—about half found no effect, a quarter found a statistically significant positive effect, and a quarter found a statistically significant negative effect—revealing that even with controlled data and responsible methodologies, social research yields scattershot conclusions that publication bias then cements into 'settled' policy-shaping wisdom.If Life Is Random, Is It Meaningless? (with Brian Klaas)
- Scientific journals like Science and Nature favor findings that make a big splash, while negative findings that help avoid mistakes rarely get published, which distorts the evidence base.Lex Fridman Podcast #502 - Psychiatry, Insane Asylums, Mental Illness, ECT, Lobotomies, Freud & Jung
- Science needs a journal of insignificant results, because the failure to publish null findings makes the research process look like a seamless straight line when it was full of zigzags.William Byers on the Blind Spot, Science, and Uncertainty 05/16/2011
- Publication bias (a form of confirmation bias) is a pervasive problem across all research fields — economics, medicine, biology, chemistry — because non-significant or non-confirming results are discarded or never published, and researchers may run multiple trials and keep only the wanted result.Daniel Pink on Drive, Motivation, and Incentives 08/30/2010
- The natural incentive to hide uninteresting or null results means the studies we observe disproportionately come from the tails of the distribution of possible results, making the published literature less reliable, as seen in the replication crisis in psychology and beyond.Julia Belluz on Epidemiology, Nutrition, and Metabolism
- The decline effect—where effects shrink upon re-testing—is most plausibly explained by the joint operation of publication bias requiring p<.05 and pervasively underpowered research: with low power, only chance-inflated, larger-than-real effects clear the significance threshold and get published, so subsequent unbiased replications estimate smaller, truer effects.Brian Nosek on the Reproducibility Project
- Many small-group studies in social psychology and economics fail to replicate, not necessarily through fraud but because researchers seeking particular results make choices and face publication bias that push toward positive findings, so prosocial-happiness results should be treated with skepticism.Emiliana Simon-Thomas on Happiness
- Pre-registration of hypotheses and methods with peer review before conducting experiments, with guarantee of publication regardless of results so long as procedures are followed, eliminates publication bias, promotes higher powered studies, and lessens incentives for p-hacking.Is Most Published Research Wrong?