What this covers

Adam Mastroianni makes a case against peer review as a fundamentally broken system that should be abandoned rather than reformed. Speaking with Russ Roberts on EconTalk, he argues that the modern centralized journal system—a 20th-century invention—confers an unwarranted "imprimatur of truth" on papers without actually checking them, making it worse than no gatekeeping at all. The conversation moves between historical perspective, empirical evidence of review's failure, and a theoretical case for why science would thrive under a more open ecosystem instead.

The empirical foundation rests on studies showing peer review rarely catches errors: research by Richard Smith at the British Medical Journal found reviewers detected only about 25% of deliberately inserted flaws, even missing major problems like nonrandom randomization in supposedly randomized controlled trials. Mastroianni notes that reviewers almost never actually inspect data or rerun analyses—they offer aesthetic judgment instead. Fraudulent papers get caught only after publication when someone independently decides to investigate. He cites prominent cases including a fabricated PNAS study on signature placement and the vaccine-autism scare fueled by a weak Lancet paper with eighteen subjects, both of which gained credibility precisely from journal prestige.

The theoretical turn invokes strong-link versus weak-link problems: science advances at the rate of its best work while poor work falls away, so suppressing variance costs more than it saves. Mastroianni argues the opportunity cost is severe—reviewers spend an estimated 15,000 person-years annually—while journals sort work mainly for status rather than serving knowledge users. He draws contrasts to earlier scientific communication through letters and magazines, and notes Albert Einstein published outside peer review when possible. The case culminates in a call to abandon the system and let emergent order replace centralized vetting.

Sharpest takeaway

Mastroianni argues that peer review is a recent, failed experiment that doesn't actually check papers for truth yet confers unwarranted trust, and that science—being a strong-link problem—would be better served by an open, diverse ecosystem of publishing than by trying to fix peer review.

  • Peer review rarely checks data or analysis and catches only a small fraction of deliberately inserted major errors
  • The system confers an 'imprimatur of truth' that misleads consumers, making it worse than nothing
  • Science is a strong-link problem where the best work matters and bad work falls away, so suppressing variance costs more than it saves

The argument · threads4 threads · 29 claims
0.78

Peer review does not reliably ensure published results are true or replicable.

3 pointscentrality 5/5
  • Due to publication biases and academic incentives, many results published in peer-reviewed journals fail to replicate when actually tested—fraud being only an extreme case—so being published is not the same as being true.

    “Published and true are not synonyms.”

  • Daniel Kahneman admitted he placed too much faith in underpowered priming studies in Thinking Fast and Slow, writing that scientific norms required believing a conclusion supported by a large body of evidence in reputable journals even when initially implausible—demonstrating that even a Nobel laureate deferred to peer-reviewed publication over his own skepticism.

    “My position when I wrote Thinking Fast and Slow was that if a large body of evidence published in reputable journals supports an initially implausible conclusion, then scientific norms require us to believe that conclusion.”

  • Replicating another scientist's paper has historically been nearly worthless professionally: if the result replicates there is nothing new to publish, and if it fails to replicate the pursuit still isn't prestigious, which is why so few results were ever checked until projects deliberately created incentives for it.

    “Replicating someone else's paper is almost worthless historically in over the last 50 years of this process... if you find out that it is true, nobody wants to publish it. There's nothing new there.”

0.57

Open publishing without peer review could produce better scientific discovery through emergent norms and institutions.

4 pointscentrality 3/5
  • From a Hayekian view of emergent order, if more scientists abandoned peer review in favor of open publishing, new institutions and norms would emerge from the bottom up to organize discovery—an ecosystem of exploration like the pre-journal era that might work even better given modern tools, though the oversupply of PhDs is a complicating difference.

    “as a Hayekian believer in emergent order, I think if more people felt the way you did, other institutions, norms, and other things would emerge that--from the bottom up”

  • Only one of Einstein's papers ever went through peer review, and it annoyed him so much when he learned the editor had sent it out that he published it elsewhere—illustrating that great science was historically done without the modern peer review system.

    “only one of Einstein's papers was peer reviewed, and it annoyed him so much he published it somewhere else when he found out the editor had sent it out”

  • Whereas science is a strong-link problem where the marketplace of ideas eventually judges bad work harshly and resiliently, social media may be a weak-link problem where unfiltered conspiracy claims believed by many can be genuinely harmful because the incentives and reputational stakes that discipline science are absent.

    “it seems like on Twitter, Facebook--social media generally--that it might be a weak-link problem. Someone who is poking around making wild conspiracy claims that are believed by a lot of people is maybe not so healthy.”

  • Peer review is analogous to a king having a taster—or two tasters—check food for poison before he eats: a system designed to catch danger before it reaches the consumer of knowledge.

    “You could argue it's kind of like how the king might have a taster.”

0.49

More information alone does not improve decisions without the capacity to properly assess and weigh it.

1 pointcentrality 4/5
  • The widespread belief that more information always leads to better decisions rests on a false assumption: more information is better only if you can assess and weigh it properly; if you are overconfident in imperfect information, additional information leads you to a different kind of error rather than a better decision.

    “More information is better only if you can weigh it properly. Only if you can assess it properly. If you overreact to it, if you are overly confident in the information, the imperfect information you get, you make a different kind of error.”

0.49

Modern publish-or-perish incentives push scholars toward safe, trivial work rather than bold, important research.

1 pointcentrality 4/5
  • The modern incentive of publish-or-perish, combined with too many PhDs and risk-aversion, pushes scholars to work on tiny, safe questions that reliably reach statistical significance but are unimportant to the world—a worse tragedy than the wasted reviewing hours—whereas the pre-journal era's smaller pool of bolder risk-takers with low material returns produced different, riskier work.

    “there's a huge incentive to publish. And, that quote changes everything... It pushes you towards smaller thoughts and smaller questions because the return is more--safer.”