causal

Multiple Testing Requires Higher Significance Bar

When the number of tests increases, the standard two-sigma (95% confidence) rule becomes unreliable because random chance will produce apparent significant findings; with 20 tests you expect roughly one false positive at the 5% level, so the cutoff must be raised as in physics' five-sigma standard.

causalpending

Speaker

Campbell Harvey

Evidence Quote

“when you do 20 tests, you can't use the 2-sigma rule. So by the 2-sigma rule, if you try 20 things, then the odds are that something is going to come up as a fluke”

Source

Campbell Harvey on Randomness, Skill, and Investment Strategies— EconTalk
Created: 6/13/2026, 12:29:08 AM

Your notes

Loading notes...