statistical significance
A measure of whether an observed effect is unlikely to be due to chance; the Oregon death reduction was real in size but not statistically significant.
As defined in Atul Gawande
Where it comes up · 18
- Andrew Gelman on Social Science, Small Samples, and the Garden of the Forking Paths9 claims
- Is Most Published Research Wrong?8 claims
- Jim Manzi on the Oregon Medicaid Study, Experimental Evidence, and Causality3 claims
- Betsey Stevenson and Justin Wolfers on Happiness, Growth, and the Reinhart-Rogoff Controversy3 claims
- How to Avoid Lying With Statistics (with Jeremy Weber)3 claims
- Brian Nosek on the Reproducibility Project2 claims
- Tim Harford on the Virtues of Disorder and Messy2 claims
- Atul Gawande1 claim
and 10 more
Most important claims · 10 of 41Search all claims for “statistical significance” →
- Because there are many different analyses a researcher could run on any dataset and only one needs to be statistically significant to get published, the true chance of finding a 'significant' result under the null is far higher than 5%—like a lottery where you keep buying tickets until you win—even without any conscious cheating.Andrew Gelman on Social Science, Small Samples, and the Garden of the Forking Paths
- Quora enforces that every experiment either declares a hypothesis in advance or explicitly declares it a learning experiment; if a learning experiment surfaces a promising effect, a follow-up experiment must be run with that as the hypothesis, because otherwise you run into statistical significance problems.Adam D'Angelo on Knowledge, Experimentation, and Quora
- To guard against unreliable single studies, Harford avoids reporting amazingly counterintuitive results that pass significance only marginally (e.g., p=0.04) and instead builds arguments from multiple independent sources—stories, interviews with participants, social psychology, computer science, cognitive psychology, neuroscience—so that even if any single study fails to replicate, convergence from many different angles gives reasonable confidence.Tim Harford on the Virtues of Disorder and Messy
- Psychology's replication crisis stems from running many studies and publishing only the ~1-in-20 that hits 5% significance while discarding the rest, an under-reporting problem. Economics may be worse, because instead of costly lab experiments, economists can cheaply download data and run regressions, discarding non-results — making it even easier to sift variables until something is significant. The remedy is testing ideas across many datasets and treating replication as re-testing an idea in a new way, not just re-running someone's data.Betsey Stevenson and Justin Wolfers on Happiness, Growth, and the Reinhart-Rogoff Controversy
- In a small, highly variable sample the standard error is large, so any result that can reach statistical significance must be large (about two standard errors from zero); because only large estimates clear the significance bar, published significant effects are systematically overestimated due to selection bias.Andrew Gelman on Social Science, Small Samples, and the Garden of the Forking Paths
- Across the 100 replications, replication effects were on average half the magnitude of the original effects: 97% of original studies reported statistically significant results but only 36% of replications did, and only 39% of effects were subjectively rated as having replicated.Brian Nosek on the Reproducibility Project
- Easterlin's cross-country null finding was an artifact of small samples: failing to find a statistically significant effect, he assumed it was zero, but absence of evidence is not evidence of absence. With modern data from 160 countries, GDP per capita is robustly related to average life satisfaction, with a correlation of about 0.8 — one of the tightest cross-country relationships observed.Betsey Stevenson and Justin Wolfers on Happiness, Growth, and the Reinhart-Rogoff Controversy
- Statistical significance (whether a relationship differs from random, e.g. via the two-sigma rule) is conceptually distinct from economic significance (whether the effect is actually large enough to matter); a finding can be statistically significant yet trivially small in real-world terms.Campbell Harvey on Randomness, Skill, and Investment Strategies
- Journals are far more likely to publish results that reach statistical significance (p<0.05), creating incentives for researchers to use analytical methods that produce lower p-values, thus promoting p-hacking as a successful publication strategy.Is Most Published Research Wrong?
- Researchers can make many decisions about data analysis that decrease p-values, including: analyzing data, finding it nearly significant, collecting more data points until p<0.05 is achieved, then stopping. Numerical simulations show this produces false positives where relationships cross the significance threshold through additional data even though larger samples would show no true relationship.Is Most Published Research Wrong?