p-value
A frequentist statistical measure of how surprising data would be under a null hypothesis, often used as a threshold for significance.
As defined in ms_ama_october_2024
Other definitions
In statistical testing, the probability of observing results as extreme as those actually observed if the null hypothesis were true; conventionally, p < 0.05 is considered statistically significant
The Market is Getting Less Efficient: Here is Why That is GREAT for Value Investors | Cliff AsnessProbability of observing results at least as extreme as the data under the null hypothesis; a conventional threshold of 0.05 is used to claim statistical significance.
Biotech’s Chief Evangelist
Where it comes up · 14
- Is Most Published Research Wrong?6 claims
- John Ioannidis on Statistical Significance, Economics, and Replication6 claims
- Brian Nosek on the Reproducibility Project4 claims
- Andrew Gelman on Social Science, Small Samples, and the Garden of the Forking Paths4 claims
- Quant vs. Traditional Investors and How Alphas Become Betas3 claims
- Jim Manzi on the Oregon Medicaid Study, Experimental Evidence, and Causality2 claims
- Campbell Harvey on Randomness, Skill, and Investment Strategies2 claims
- A conversation between Nassim Nicholas Taleb and Stephen Wolfram at the Wolfram Summer School 20211 claim
and 6 more
Most important claims · 10 of 34Search all claims for “p-value” →
- When the number of tests increases, the standard two-sigma (95% confidence) rule becomes unreliable because random chance will produce apparent significant findings; with 20 tests you expect roughly one false positive at the 5% level, so the cutoff must be raised as in physics' five-sigma standard.Campbell Harvey on Randomness, Skill, and Investment Strategies
- In Table 5, the estimated causal effect of Medicaid coverage was an increase in smoking incidence, from about 43% in the control group to about 48% in the test group, a five-percentage-point increase that was not statistically significant (p-value about 0.18) but had a closer p-value than nearly any other result in the study.Jim Manzi on the Oregon Medicaid Study, Experimental Evidence, and Causality
- Statistical significance (whether a relationship differs from random, e.g. via the two-sigma rule) is conceptually distinct from economic significance (whether the effect is actually large enough to matter); a finding can be statistically significant yet trivially small in real-world terms.Campbell Harvey on Randomness, Skill, and Investment Strategies
- Journals are far more likely to publish results that reach statistical significance (p<0.05), creating incentives for researchers to use analytical methods that produce lower p-values, thus promoting p-hacking as a successful publication strategy.Is Most Published Research Wrong?
- Many of the biases that operate in research share a common denominator: people are trying to find something rather than trying not to find something, leading them to change analyses, add observations, or keep trying until they get a statistically significant p-value, which—when added to low power—causes false-positive rates and effect exaggeration to skyrocket.John Ioannidis on Statistical Significance, Economics, and Replication
- Researchers can make many decisions about data analysis that decrease p-values, including: analyzing data, finding it nearly significant, collecting more data points until p<0.05 is achieved, then stopping. Numerical simulations show this produces false positives where relationships cross the significance threshold through additional data even though larger samples would show no true relationship.Is Most Published Research Wrong?
- Statistical significance does not mean a result is almost certainly not due to noise; it only means that IF there were nothing but noise, the chance of seeing something as extreme or greater is less than 1 in 20—a conditional statement people routinely garble.Andrew Gelman on Social Science, Small Samples, and the Garden of the Forking Paths
- In a realistic scenario with 1000 hypotheses being tested (10% true, 90% false), with 80% statistical power to detect true effects and using p<0.05 threshold, approximately 80 true positive results and 45 false positive results will be published, meaning nearly one-third (approximately 36%) of published results will be false even though the statistical system is working normally.Is Most Published Research Wrong?
- In statistics 'significant' means a result is unlikely to have occurred by chance, which is distinct from the everyday meaning of importance; a statistically significant effect can have an effect size so small that it is practically unimportant, as with the bacon-cancer link.Brian Nosek on the Reproducibility Project
- The p-value threshold of 0.05 for statistical significance was arbitrarily selected by Ronald Fisher in a book published in 1925, rather than being grounded in a principled rationale.Is Most Published Research Wrong?