What this covers

Derek Muller builds the video around a puzzle: a 2011 psychology paper, "Feeling the Future," reported that participants could predict future images at better than chance, a 53% hit rate on erotic images against 50% expected, with a p-value of 0.01. Journals published it, and a later replication by three researchers was turned down by the same journal because it doesn't run replications. From there he works out why a system built on rigorous methods still yields so many wrong results.

The core is arithmetic and incentives. The 0.05 threshold for statistical significance was picked somewhat arbitrarily by Ronald Fisher in 1925. Muller walks through a scenario of 1000 hypotheses, 10% of them true, with 80% statistical power, and shows that about 36% of published positives would be false even when nothing malfunctions. He then covers how p-hacking works: collecting data until a result crosses the line, tracking many variables, and combining tactics to push false-positive rates above 60%. He stresses that most researchers aren't malicious, because analysis involves legitimate choices. A deliberately flawed chocolate weight-loss study and 29 teams analyzing the same soccer red-card data with different conclusions illustrate this.

The video also argues the problem isn't confined to soft sciences. The 2002 Theta-plus pentaquark, seen by 11 further experiments at 5-sigma, turned out to be false, which Muller attributes to researchers not being blind to the data. He explains why replication fails to self-correct science in practice, given publication bias and the career pressures Brian Nosek describes. Near the end he turns to remedies, chiefly pre-registration with guaranteed publication, plus replication projects, Retraction Watch, and repositories for negative results. His closing reflection is that science remains our most reliable way of knowing, yet still errs often.

Sharpest takeaway

The scientific publishing system has systematic incentive structures that produce a high proportion of false positive results despite rigorous statistical methods, undermining the reliability of published research across disciplines.

  • Publication bias and p-hacking incentives mean nearly one-third of published results are false even when the system works normally
  • Reproducibility studies find only 36% of psychology papers replicate and 11% of landmark cancer studies replicate, far worse than statistical theory predicts
  • Scientists face career incentives to publish novel positive results rather than conduct replications, creating a systematic bias toward false discoveries

The claims · ranked27 claims · weighted by value

This asset isn't compiled yet

You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.

0.80

Scientists have enormous incentives to publish papers because their careers depend on publications, and as researcher Brian Nosek states, 'There is no cost to getting things wrong, the cost is not getting them published'.

causalhigh valueestablishednovelty 2/4durability 4/4· Derek Muller

“scientists have huge incentives to publish papers, in fact their careers depend on it; as one scientist Brian Nosek puts it: 'There is no cost to getting things wrong, the cost is not getting them published'”

0.80

For a scientist, the successful publication strategy is to avoid replication studies (since journals rarely publish them and results are likely not significant) and instead test novel and unexpected hypotheses while p-hacking to achieve statistical significance.

causalhigh valueestablishednovelty 2/4durability 4/4· Derek Muller

“So if you're a scientist the successful strategy is clear and don't even attempt replication studies because few journals will publish them, and there is a very good chance that your results won't be statistically significant any way... So a far better approach is to test novel and unexpected hypotheses and then p-hack your way to a statistically significant result”

0.80

The p-value threshold of 0.05 for statistical significance was arbitrarily selected by Ronald Fisher in a book published in 1925, rather than being grounded in a principled rationale.

factualhigh valueestablishednovelty 2/4durability 4/4· Derek Muller

“the .05 threshold was arbitrarily selected by Ronald Fisher in a book he published in 1925”

0.80

In a realistic scenario with 1000 hypotheses being tested (10% true, 90% false), with 80% statistical power to detect true effects and using p<0.05 threshold, approximately 80 true positive results and 45 false positive results will be published, meaning nearly one-third (approximately 36%) of published results will be false even though the statistical system is working normally.

causalhigh valueestablishednovelty 2/4durability 4/4· Derek Muller

“Imagine you're a researcher in a field where there are a thousand hypotheses currently being investigated. Let's assume that ten percent of them reflect true relationships and the rest are false... assuming the experiments are pretty well designed, they should correctly identify around say 80 of the hundred true relationships... so 20 results are false negatives... from those 900 false hypotheses using a p-value of .05, forty-five false hypotheses will be incorrectly considered true... the papers that eventually get published will include 80 true positive results: 45 false positive results and maybe 20 true negative results. Nearly a third of published results will be wrong”

0.80

Researchers can make many decisions about data analysis that decrease p-values, including: analyzing data, finding it nearly significant, collecting more data points until p<0.05 is achieved, then stopping. Numerical simulations show this produces false positives where relationships cross the significance threshold through additional data even though larger samples would show no true relationship.

causalhigh valueestablishednovelty 2/4durability 4/4· Derek Muller

“Researchers can make a lot of decisions about their analysis that can decrease the p-value, for example let's say you analyze your data and you find it nearly reaches statistical significance, so you decide to collect just a few more data points to be sure then if the p-value drops below .05 you stop collecting data, confident that these additional data points could only have made the result more significant if there were really a true relationship there, but numerical simulations show that relationships can cross the significance threshold by adding more data points even though a much larger sample would show that there really is no relationship”

0.80

Most scientists are not p-hacking maliciously; rather, there are legitimate decisions to be made about how to collect, analyze and report data, and these decisions impact the statistical significance of results, so even well-intentioned researchers can produce biased findings.

causalhigh valueestablishednovelty 2/4durability 4/4· Derek Muller

“Now most scientists aren't p-hacking maliciously, there are legitimate decisions to be made about how to collect, analyze and report data, and these decisions impact on the statistical significance of results”

0.80

Publication is also more likely if results are novel and unexpected, which encourages researchers to investigate increasingly unlikely hypotheses, thereby decreasing the ratio of true to spurious relationships being tested in the literature.

causalhigh valueestablishednovelty 2/4durability 4/4· Derek Muller

“publication's also more likely if the result is novel and unexpected, this encourages researchers to investigate more and more unlikely hypotheses which further decreases the ratio of true to spurious relationships that are tested”

0.80

Journals are far more likely to publish results that reach statistical significance (p<0.05), creating incentives for researchers to use analytical methods that produce lower p-values, thus promoting p-hacking as a successful publication strategy.

causalhigh valueestablishednovelty 2/4durability 4/4· Derek Muller

“Journals are far more likely to publish results that reach statistical significance so if a method of data analysis results in a p-value less than .05 then you're likely to go with that method”

0.80

Replication studies should theoretically allow science to self-correct, but in practice they are rarely published by journals, so if a researcher attempts a replication and fails to find significance, they cannot publish the negative result and may be accused of poor methodology rather than the original finding being false.

causalhigh valueestablishednovelty 2/4durability 4/4· Derek Muller

“when they tried to publish their findings in the same journal as the original paper they were rejected. The reason? The journal refuses to publish replication studies... in which case instead of being able to convince colleagues of the lack of reproducibility of an effect you will be accused of just not doing it right”

0.75

The proportion of false results in published research gets even worse than ~33% if studies are underpowered (which they typically are), if there is a higher ratio of false-to-true hypotheses being tested, or if researchers are biased.

causalhigh valueestablishednovelty 2/4durability 3/4· Derek Muller

“things get even worse if studies are underpowered, and analysis shows they typically are, if there is a higher ratio of false-to-true hypotheses being tested or if the researchers are biased”

0.75

A 2015 study showing that eating chocolate daily helps with weight loss was deliberately designed with methodological flaws to generate false positives: the sample size was extremely small (5 people per group), and 18 different measurements were tracked per person, creating multiple opportunities for statistical significance on unrelated variables even though the primary hypothesis might not be supported.

factualhigh valueestablishednovelty 2/4durability 3/4· Derek Muller

“the whole thing had been faked, kind of. I mean researchers did perform the experiment exactly as they described, but they intentionally designed it to increase the likelihood of false positives: the sample size was incredibly small, just five people per treatment group, and for each person 18 different measurements were tracked including: weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, and so on”

0.74

The pentaquark false discovery was caused by researchers not being 'blind' to the data—they knew how the numbers were generated and what answer they expected to find, and the way the data was cut and analyzed (p-hacked) produced the false finding.

causalhigh valueestablishednovelty 1/4durability 4/4· Derek Muller

“The problem was those first scientists weren't blind to the data, they knew how the numbers were generated and what answer they expected to get, and the way the data was cut and analyzed, or p-hacked, produced the false finding”

0.73

If everyone uses p<0.05 as a cutoff for statistical significance, the intuitive expectation would be that 5% of all published results are false positives, but this grossly underestimates the actual proportion of false findings in published literature.

causalhigh valueestablishednovelty 2/4durability 4/4· Derek Muller

“The intuitive answer seems to be five percent. I mean if everyone is using p less than .05 as a cut-off for statistical significance, you would expect five of every hundred results to be false positives but that unfortunately grossly underestimates the problem”

0.73

Pre-registration of hypotheses and methods with peer review before conducting experiments, with guarantee of publication regardless of results so long as procedures are followed, eliminates publication bias, promotes higher powered studies, and lessens incentives for p-hacking.

causalhigh valueestablishednovelty 2/4durability 4/4· Derek Muller

“This eliminates publication bias, promotes higher powered studies and lessens the incentive for p-hacking”

0.73

Even trying our best to figure out what's true using scientific methods, we must acknowledge we get it wrong frequently, and the question of how much we delude ourselves when not using the scientific method becomes more troubling when we recognize that science—as flawed as it is—is far more reliable than any other way of knowing we have.

normativehigh valueestablishednovelty 2/4durability 4/4· Derek Muller

“As flawed as our science may be, it is far away more reliable than any other way of knowing that we have”

0.73

When 29 different research groups were given identical data about soccer player red cards and asked to determine if dark-skinned players are more likely to receive them, some groups found no significant effect while others concluded dark-skinned players were three times as likely to receive a red card, demonstrating that data does not speak for itself but must be interpreted.

factualhigh valueestablishednovelty 2/4durability 4/4· Derek Muller

“29 different research groups were given the same data and asked to determine if dark-skinned soccer players are more likely to be given red cards; using identical data some groups found there was no significant effect while others concluded dark-skinned players were three times as likely to receive a red card”

0.73

The most striking aspect of the reproducibility crisis is not the prevalence of incorrect information in published journals but rather that even using the most sophisticated and rigorous methods—peer review, statistical tools, and standards of practice—scientists still get it wrong frequently.

normativehigh valueestablishednovelty 2/4durability 4/4· Derek Muller

“The thing I find most striking about the reproducibility crisis in science is not the prevalence of incorrect information in published scientific journals after all getting to the truth we know is hard and mathematically not everything that is published can be correct. What gets me is the thought that even trying our best to figure out what's true, using our most sophisticated and rigorous mathematical tools: peer review, and the standards of practice, we still get it wrong so often”

0.70

A p-value is only really valid for testing a single measure; once multiple variables are being compared, the probability that at least one of them produces a false positive increases substantially, a phenomenon known as 'p-hacking' or the multiple comparisons problem.

causalhigh valueestablishednovelty 1/4durability 4/4· Derek Muller

“The point is: a p-value is only really valid for a single measure once you're comparing a whole slew of variables the probability that at least one of them gives you a false positive goes way up, and this is known as 'p-hacking'”

0.69

A 2011 paper in the Journal of Personality and Social Psychology titled 'Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect' reported finding that participants could predict future images at rates significantly better than chance, with a hit rate of 53% for erotic images compared to 50% expected by random guessing, yielding a p-value of 0.01.

factualhigh valueestablishednovelty 1/4durability 3/4· Derek Muller

“In 2011 an article was published in the reputable 'Journal of Personality and Social Psychology'. It was called 'Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect' or, in other words, proof that people can see into the future.”

0.69

Combining three p-hacking strategies together (having two dependent variables, adding more observations, controlling for gender, or dropping one of three conditions) increases the likelihood of a false positive to over 60% when using p<0.05 as the threshold.

factualhigh valueestablishednovelty 1/4durability 3/4· Derek Muller

“there are a great number of ways to increase the likelihood of significant results like: having two dependent variables, adding more observations, controlling for gender, or dropping one of three conditions combining all three of these strategies together increases the likelihood of a false-positive to over sixty percent, and that is using p less than .05”

0.69

The problem of false positives through p-hacking is not limited to soft sciences like psychology and neuroscience; it also appears in physics, as demonstrated by the 2002 Japanese discovery of the Theta-plus pentaquark, an exotic particle made of five quarks, which was subsequently found by 11 other experiments but ultimately proven false despite using 5-sigma statistical significance (one chance in 3.5 million of false positive).

factualhigh valueestablishednovelty 1/4durability 3/4· Derek Muller

“Particle physics employs particularly stringent requirements for statistical significance referred to as 5-sigma or one chance in 3.5 million of getting a false positive, but in 2002 a Japanese experiment found evidence for the Theta-plus pentaquark, and in the two years that followed 11 other independent experiments then looked for and found evidence of that same pentaquark with very high levels of statistical significance... but alas, it was a false discovery for their experimental attempts to confirm that theta-plus pentaquark using greater statistical power failed to find any trace of its existence”

0.69

Over the past 10 years, improvements have been made to the reproducibility crisis, including: more large-scale replication studies being undertaken, Retraction Watch publicizing withdrawn papers, online repositories for unpublished negative results, and pre-registration of hypotheses and methods with guarantee of publication if procedures are followed regardless of results.

factualhigh valueestablishednovelty 1/4durability 3/4· Derek Muller

“over the past 10 years things have started changing for the better. Many scientists acknowledge the problems i've outlined and are starting to take steps to correct them: there are more large-scale replication studies undertaken in the last 10 years, plus there's a site, Retraction Watch, dedicated to publicizing papers that have been withdrawn, there are online repositories for unpublished negative results and there is a move towards submitting hypotheses and methods for peer review before conducting experiments with the guarantee that research will be published regardless of results so long as the procedure is followed”

0.66

A 2005 paper entitled 'Why Most Published Research is False' pointed out the mathematical relationship between false and true hypotheses, statistical power, and publication bias that explains high false positive rates in published research.

factualhigh valueestablishednovelty 1/4durability 4/4· Derek Muller

“All of this was pointed out in 2005 paper entitled 'Why most published research is false'”

0.65

The precognition study from the beginning of this video was replicated by three researchers who found no significant effect different from chance, but when they tried to publish in the same journal they were rejected because the journal refuses to publish replication studies.

factualhigh valueestablishednovelty 1/4durability 3/4· Derek Muller

“three researchers attempted to replicate one of those experiments, and what did they find? well, surprise surprise, the hit rate they obtained was not significantly different from chance. When they tried to publish their findings in the same journal as the original paper they were rejected. The reason? The journal refuses to publish replication studies”

0.61

The Reproducibility Project attempted to replicate 100 psychology studies and found that only 36% had a statistically significant result the second time, and the strength of measured relationships was on average half those of the original studies.

factualhigh valueestablishednovelty 1/4durability 3/4· Derek Muller

“The Reproducibility Project repeated a hundred psychology studies but found only thirty-six percent had a statistically significant result the second time around and the strength of measured relationships were on average half those of the original studies”

0.61

The chocolate weight loss study received widespread media coverage including front pages in Bild (most widely circulated daily newspaper in Europe), the Daily Star, Irish Examiner, Huffington Post, and Shape Magazine, despite the underlying research being deliberately flawed to demonstrate p-hacking.

factualhigh valueestablishednovelty 1/4durability 3/4· Derek Muller

“As you might expect this news spread like wildfire, to the front page of Bild, the most widely circulated daily newspaper in Europe and into the Daily Star, the Irish Examiner, to Huffington Post and even Shape Magazine”

0.61

An attempted verification of 53 studies considered landmarks in the basic science of cancer only managed to reproduce 6, even when working closely with the original study authors, which is worse than the theoretical prediction of false positive rates would suggest.

factualhigh valueestablishednovelty 1/4durability 3/4· Derek Muller

“An attempted verification of 53 studies considered landmarks in the basic science of cancer only managed to reproduce six even working closely with the original study's authors these results are even worse than i just calculated”