Brian Nosek on Truth, Science, and Academic Incentives
What this covers
Brian Nosek of the University of Virginia joins Russ Roberts for a conversation centered on a single tension: academic incentives reward publishable results, but publishable and true are not the same thing. Nosek argues that this misalignment does not require conscious fraud to corrupt the scientific record. Instead, honest researchers face researcher degrees of freedom—many defensible choices about which participants to exclude, which covariates to include, which analyses to run—and subtle pressure to choose whichever path produces the most novel or significant finding. The cumulative effect is a literature skewed toward false positives. The conversation then turns practical: rather than waiting for journals or universities to overturn their reward structures, Nosek sketches targeted changes that realign scientists' everyday incentives toward verification without dismantling the drive to innovate. These include pre-registration of analysis plans before data access, specialized venues for replication work, and open workflows that make researcher choices reviewable after the fact.
The discussion ranges across psychology, economics, and medicine, with concrete examples: industrial labs at Bayer and Amgen attempting to replicate academic findings found replication rates near 11–25 percent; a high-profile study on political extremism and visual perception failed to reproduce. Nosek and Roberts spar on where the problem sits sharpest—Roberts contends that fields like epidemiology amount to data fishing expeditions lacking mechanistic grounding, while Nosek acknowledges the risk but distinguishes it from the separate hazard of researcher degrees of freedom. They examine Publication Bias and the file drawer problem, the mismatch between what gets published and what actually happened in labs, and consider whether Innovation vs Verification Balance—how much journal space should go to confirmation rather than novelty—can shift without crippling scientific ambition. Throughout, the speakers treat self-correction not as a flaw but as science's distinguishing feature, though one that works too slowly when incentives are misaligned.
Nosek argues that academic incentives reward publishable, novel results over true ones, subtly biasing even honest researchers toward false positives, and that the fix is building practical tools (registration, replication outlets, open workflows) that realign scientists' everyday incentives toward verification.
- Published and true are not synonyms; publication incentives conflict with truth-seeking
- Researcher degrees of freedom let chance produce favorable results without conscious fraud
- Self-correction is too slow; targeted incentive tweaks (Open Science Framework, replication issues, PLOS One) can speed it
Journal and institutional incentives must shift toward valuing verification by integrating replication into existing prestige structures and everyday practice.
- Two industrial laboratories (Bayer and Amgen) attempting to replicate academic results as a first step toward drug development found stunningly low replication rates—about 11% in one effort and roughly 25% in another—because they have strong financial incentives not to waste money developing drugs based on unsound research.
“One of the studies, it was 11% of the results replicated; another it was about 25%. And those are stunningly low.”
- Journals chase prestige and attention, so they reward innovation—pushing at the boundaries of knowledge, which by definition entails risk and being wrong—while undervaluing the boring but equally important work of confirmation and verification; the fix is to rebalance incentives enough to encourage verification of provocative findings without flipping entirely to verification.
“we are strongly tilted in scientific discipline to value the first one, the innovation part, at the expense of the verification part”
- Journals defined solely by publishing null results or replications repeatedly fail because being defined as the outlet for things no other journal would publish signals a crummy journal, giving individual scientists no incentive to submit; solutions must instead integrate into existing prestigious publication structures.
“they are journals that are defined by the fact that they are publishing things that no other journals would publish. Which means: We are a crummy journal.”
- Science is self-correcting only eventually, meaning over a very long time, which wastes resources and time believing things that could be cleared up much faster; modest tweaks to incentives that make verification part of ordinary practice could accelerate self-correction on important findings.
“when we use the common trope of "science is self-correcting," when we talk about eventually we mean a very, very long time. And that's not okay with me.”
- Publishing replications of high-impact results in a special issue of a mid-tier journal aligns incentives for both parties: the researcher gets a publication, and the journal gains relatively strong citations because it is republishing results already known to be important—creating a small nudge to make replication more normative without overwhelming the system.
“the replications will get relatively strong citations, too, because they are publishing results that are already known to be important results. So it makes it for the journal something that is of value.”
- PLOS One's review process judges only the soundness of the research—whether the design is well conducted and the interpretations justified—without regard to importance or novelty, which opens the door to publishing replications and null results and helps shift incentives away from privileging only exciting findings.
“The review process is solely about the soundness of the research... Without any consideration of importance.”
- Fixing journals alone is insufficient because universities face the same prestige incentives: a research university that declares publishing unimportant would be defining itself out of being a research university, so the most tractable lever is realigning the everyday individual practices of scientists rather than top-down institutional reform.
“if a university says: Okay, well, publishing isn't really important for us--then they are really defining themselves as not being a research university.”
Publication bias toward novel, positive results leaves the true state of the literature unknown and unmeasured.
- Despite many reasons to worry that the research literature is biased, there is little empirical evidence in economics, psychology, or any discipline estimating how reproducible the science actually is—we have no clear idea how biased the literature is, which is the most surprising gap.
“we have lots of worries, lots of reasons to have worries... But we just really have no clue how biased the research literature is.”
- Because positive and innovative results are more publishable than negative or confirmatory ones, the unpublished 'file drawer' (gray literature) systematically differs from the published record, so knowing what is in both literatures is essential to understanding what is really going on.
“It's a biased representation in the published literature than what is in the unpublished literature. And so knowing what's in both of the literatures is very useful to understanding what's going on.”
- The incentive to gain fame by refuting a result only applies to very famous findings; merely influential results are not worth anyone's time to replicate because debunking them earns no attention, and obscure results that influence no one are not worth replicating at all—so verification effort tracks where a result sits on the fame continuum.
“There is some of that, but it's really there for things that get really famous. It's not there for stuff that's just kind of influential.”
- PsychFileDrawer, started by Hal Pashler and colleagues, lets researchers post a short 15-minute summary of unpublished replication results, providing a low-cost repository to expose the file drawer and connect researchers interested in the same findings without the burden of formal publication.
“you can write up a very short summary, takes 15 minutes, to post to Psych File Drawer what you found in this replication project.”
Unconstrained researcher choices in analysis and data handling allow chance to manufacture publishable results that fail replication.
- Running many low-powered studies or trying many specifications and then reporting only the final result that passes p < .05 makes the standard significance criterion irrelevant, because the cumulative chances of finding an effect are not accounted for in the single reported test.
“you are just, you are adding up all of these different chances that you have to find an effect, and then still using the same criterion: p less than .05 just on the one final one you report.”
- Even without conscious fraud, researchers make many analysis decisions (which participants to exclude, which covariates to include) and are subtly influenced to choose those that make results look better; these unconstrained choices leverage chance to produce favorable results, which is why a constrained replication using the same fixed choices often fails to reproduce the original effect.
“When we can make all of these different decisions of what should I exclude and what other things should I do, we are leveraging chance. We are taking our opportunities to get results that look good for us.”
- Having two camps with opposing perspectives jointly agree on an analysis script before running it surfaces exactly where they differ—on which control variables and data exclusions to use—revealing that the real substance of scientific debate often lies in how you reach the outcome rather than in the outcome itself.
“It may not actually be in the outcomes. It may be in how you get to the outcomes. And then that's really where the substance of the debate needs to be.”
Failed replications require investigation of nuances and moderating variables rather than simple rejection of original findings.
- Apparently contradictory findings (e.g., caffeine being good and bad for you) can both be true under different constraints, and identifying the moderating or mediating variables that govern when each holds is a genuine theoretical challenge—distinct from the separate problem of analysis choices manufacturing publishable results.
“Caffeine can be good for you and bad for you. Probably both are true, given certain constraints. And part of the scientific process is identifying what those constraints are.”
- An initial study found political moderates more accurate than extremists at matching shades of gray (suggesting extremists literally see the world more black and white), but a high-powered replication with a similar sample produced no effect at all, casting doubt on the original result.
“It did not replicate. We didn't get the result again.”
- A failed replication does not unambiguously mean the original result is false; it could also reflect subtle but important and unreported nuances in procedure, materials, timing, or expertise, so systematically building cumulative science requires identifying predictors of reproducibility separate from the original being false, including engaging the original researchers before data collection.
“Failing to replicate doesn't mean that the original result is false. It could mean that, but it doesn't unambiguously mean that.”
Pre-registration and workflow documentation reduce researcher degrees of freedom and can alter behavior even without external enforcement.
- Documenting the research workflow has value even if no one ever watches it, because the mere fact that someone could review the analysis—and discover how many hours were spent fishing for a single result—can change researchers' behavior.
“even if no one else watches it, the fact that someone else could watch it could be something that changes people's behavior.”
- For strong confirmatory tests, writing up and registering the full hypothesis and analysis plan before accessing the data dramatically reduces researcher degrees of freedom; this is appropriate only for confirmatory work, not for genuinely exploratory research where no strong prediction exists.
“if there is a strong one, then registering in advance reduces your degrees of freedom dramatically”
Science's self-critical capacity and eventual self-correction are signs of health, though acceleration requires deliberate practice change.
- Science is one of the only ways of knowing that is consistently and actively self-critical, examining its own practices, identifying problems, and devising new solutions—so the current wave of scrutiny over research practices is a sign of health rather than crisis.
“Science is one of the only ways of knowing that is consistently and actively self-critical.”
- Embodiment research in psychology holds that many social concepts can have a physical basis in everyday activity and physiology, which motivated the hypothesis that political extremism might literally affect visual perception of grayscale.
“many of our social concepts can actually have a physical basis in our everyday activity and our physiology”