statistical power
The ability of an experiment to detect an effect of a given size, analogous to a microscope's magnification; higher power means smaller true effects can be reliably distinguished from chance.
As defined in Jim Manzi on the Oregon Medicaid Study, Experimental Evidence, and Causality
Where it comes up · 7
- John Ioannidis on Statistical Significance, Economics, and Replication7 claims
- Brian Nosek on the Reproducibility Project4 claims
- Is Most Published Research Wrong?3 claims
- Austin Frakt on Medicaid and the Oregon Medicaid Study3 claims
- Brian Nosek on Truth, Science, and Academic Incentives2 claims
- Jim Manzi on the Oregon Medicaid Study, Experimental Evidence, and Causality2 claims
- Raj Chetty1 claim
Most important claims · 10 of 22Search all claims for “statistical power” →
- No rational person should base a decision on undersampled results: when a study lacks enough sample for a question, its reported estimate is not much better than noise even though it has some sample, and whether a study is underpowered is an objective, computable property of any study.Austin Frakt on Medicaid and the Oregon Medicaid Study
- In a realistic scenario with 1000 hypotheses being tested (10% true, 90% false), with 80% statistical power to detect true effects and using p<0.05 threshold, approximately 80 true positive results and 45 false positive results will be published, meaning nearly one-third (approximately 36%) of published results will be false even though the statistical system is working normally.Is Most Published Research Wrong?
- Operating in an underpowered environment creates two problems simultaneously: a very high chance of false negatives (missing real signals), and a high risk that any statistically significant result is a false positive or a grossly inflated effect size—the smaller the power, the greater the average inflation of detected effects.John Ioannidis on Statistical Significance, Economics, and Replication
- Underpowered studies that do detect a signal produce dramatic papers with huge but unreliable effect estimates, an error Andrew Gelman calls a Type M error, where M stands for magnitude.John Ioannidis on Statistical Significance, Economics, and Replication
- In an underpowered environment with very small samples, a true signal can only be detected if it happens to present itself at a much bigger magnitude than reality—because if it appeared at its true size it wouldn't be significant—so any detected result is either a false positive or an exaggerated effect, a phenomenon called the Winner's Curse.John Ioannidis on Statistical Significance, Economics, and Replication
- The decline effect—where effects shrink upon re-testing—is most plausibly explained by the joint operation of publication bias requiring p<.05 and pervasively underpowered research: with low power, only chance-inflated, larger-than-real effects clear the significance threshold and get published, so subsequent unbiased replications estimate smaller, truer effects.Brian Nosek on the Reproducibility Project
- The Oregon study was statistically underpowered to detect realistic Medicaid effects on physical health: the effect sizes it could have detected were enormous (e.g., it would have needed an 80% reduction in elevated blood sugar versus the observed 20% point estimate), so the non-significant results are genuinely uninformative rather than evidence of zero effect.Austin Frakt on Medicaid and the Oregon Medicaid Study
- Using New York City data with 2.5 million observations, teacher effectiveness improves most in the first two years of experience and then follows a roughly flat profile; the original above/below-10-years split in the kindergarten STAR study was merely a statistical convenience for a small sample, not evidence that experience matters only after a decade.Raj Chetty
- Findings in cognitive psychology—basic operations of the mind like perception, memory, and attention—replicated at roughly twice the rate of social psychology findings, possibly because social effects are context-dependent and replication necessarily changes context, or because cognitive research more often uses higher-powered within-subjects designs while social psychology relies on lower-powered between-subjects designs.Brian Nosek on the Reproducibility Project
- Big data studies have the opposite problem of old small studies: they are overpowered, so nominal statistical significance means almost nothing because virtually any analysis will be significant one way or another, leaving significance with little ability to discriminate real findings from flukes.John Ioannidis on Statistical Significance, Economics, and Replication