What this covers

Campbell Harvey of Duke University sits down with Russ Roberts to examine why the standard statistical threshold used across finance and science—the two sigma rule—systematically misidentifies luck as skill. Harvey's core argument is that when researchers run many tests, some will inevitably appear significant by pure chance alone, yet the field treats any result crossing the traditional 95% confidence line as a genuine finding. He demonstrates this through concrete examples: a simulation generating 200 random trading strategies produces strategies appearing to beat the market at 2.5-sigma purely through statistical noise, and a published paper finding one of 26 company names statistically linked to stock performance was simply harvesting randomness. The conversation ranges across the mechanisms that hide this problem—hidden researcher choices in sample dates and variable selection, survivorship selection that masks a zero-skill strategy as prophetic, and the "winner's curse" where published effects shrink dramatically on replication.

Beyond counting tests, Harvey argues that standard metrics like the Sharpe Ratio hide what matters most: catastrophic downside risk. A strategy selling out-of-the-money options collects premium that looks like outperformance in calm years while creating dangerous skewness concentrated in tail events. The solution, Harvey insists, demands raising the bar to five sigma significance, measuring Fat Tails / Black Swan risk explicitly, and insisting on a First Principles Thinking economic mechanism before accepting a finding. Roberts raises the investment practitioner's perspective—that reputation incentives actually constrain data mining in money management, unlike academia where replication is rare and editors never observe the hidden choices preceding publication. The discussion covers why even Warren Buffett's long track record cannot prove skill without understanding causal logic, and why macroeconomic regression analysis remains fundamentally unreliable.

Sharpest takeaway

Harvey argues that standard statistical significance (the 2-sigma rule) systematically overstates investment skill and scientific findings because it ignores the number of tests run, so distinguishing luck from skill requires raising the significance bar, accounting for tail risk, and demanding a causal economic mechanism.

  • When many tests are run, some will appear significant by pure chance, so the 2-sigma rule must be tightened
  • The Sharpe Ratio ignores skewness and tail behavior, masking catastrophic downside risk
  • Data-mined findings need a first-principles economic story, not an ex-post concocted theory

The argument · threads5 threads · 27 claims
0.85

Multiple testing and researcher discretion create spurious findings that statistical adjustment must address.

5 pointscentrality 5/5
  • When the number of tests increases, the standard two-sigma (95% confidence) rule becomes unreliable because random chance will produce apparent significant findings; with 20 tests you expect roughly one false positive at the 5% level, so the cutoff must be raised as in physics' five-sigma standard.

    “So that's why, when you do 20 tests, you can't use the 2-sigma rule. So by the 2-sigma rule, if you try 20 things, then the odds are that something is going to come up as a fluke”

  • A circulating paper sorting stock performance by the first letter of the company name found one of 26 letters to be statistically significant, which is exactly what the jellybean multiple-testing problem predicts and demonstrates how researchers fail to adjust for data mining.

    “There's a paper that's circulating that looks at the performance of stocks sorted by the first letter of the stock name... And one of them is significant.”

  • A researcher at a top investment bank presented the 17th monthly lag of US Industrial Production as a significant market predictor, but admitted it was 'the only one that worked' out of roughly 24 lag tests, so adjusting significance for the number of tests reveals it as insignificant—a finance version of the jellybean problem.

    “this variable was the 17th monthly lag in U.S. Industrial Production... And then he basically said, 'Well, that's the only one that worked.' So, it's the jelly bean example.”

  • Spurious findings arise not just from the number of tests but from the full set of discretionary researcher choices—sample start date, trimming outliers, choice of estimation method—and because journal editors and investors never see these hidden choices, the problem is a serious and largely invisible source of false results.

    “it's not just the number of tests, but it's all the choices that researchers make. And it is a very serious problem in academic research, because the editors of the scientific journals don't see all of the choices that have been made.”

  • Harvey demonstrated that a strategy looking highly attractive at 2.5-sigma was generated from random numbers with zero true skill: simulating 200 random strategies (mean zero, S&P-like volatility), the tails inevitably produced one strategy that appeared 2.5-sigma above zero and another 2.5-sigma below, proving that apparent significance can be entirely a product of selecting from many random series.

    “what appeared to be a great strategy, was purely generated by random numbers, had nothing to do with beating the market... Every single one of these strategies by definition had zero skill. Because I fixed the return, when I'm simulating the numbers, to have an average of zero.”

0.79

Distinguishing skill from randomness requires understanding causal mechanism, not statistical significance alone.

4 pointscentrality 5/5
  • With over 10,000 money managers, some will beat their benchmark for many years in a row purely by chance, so a long track record alone (even like Warren Buffett's) cannot by itself prove skill any more than monkeys throwing darts at the Wall Street Journal.

    “when you've got 10,000 or more managers that people are going to look really good, year after year, just because you've got over 10,000 managers, purely by chance. So, they could be monkeys throwing darts at the Wall Street Journal.”

  • The first person to publish a discovery often suffers a 'winner's curse': because of the data mining that preceded publication, the reported effect is likely overstated or nonexistent, so replications typically find a much smaller or absent effect.

    “the first person to publish it, given all of the data mining that actually went on, it's likely that the effect is overstated, number one, or number two, doesn't exist. So when people replicate the study, they find that the effect is a lot smaller.”

  • Because randomness cuts both ways, a genuinely skilled manager can underperform for two or three years purely through bad luck, so investors who ditch managers after a few bad years may be making a mistake by mistaking bad luck for lack of skill.

    “often your manager doesn't perform as well as you wanted, for, let's say, 2 or 3 years in a row. And then you ditch that manager. And that's a mistake. Because it's possible that that manager is a skilled manager and basically suffered from some bad luck.”

  • Assessing whether an outcome reflects skill or randomness ultimately requires understanding the underlying causal mechanism or economic story; statistical adjustment alone is insufficient because a finding that makes no economic sense (like the 17th lag of Industrial Production or unexplained alcohol-cancer links) should be treated with deep suspicion regardless of its t-statistic.

    “the best thing you can do is to ask the question: 'What is the economic mechanism? Why does this work? Tell me the line of causality.'... you do need to have a solid economic foundation. You need to have a story. Or you should be very suspicious of the performance of a particular strategy.”

0.71

Statistical significance divorced from economic reality and tail risk creates dangerous illusions of outperformance.

5 pointscentrality 4/5
  • Statistical significance (whether a relationship differs from random, e.g. via the two-sigma rule) is conceptually distinct from economic significance (whether the effect is actually large enough to matter); a finding can be statistically significant yet trivially small in real-world terms.

    “We're talking about statistical significance by a two sigma rule. There's another concept that's equally important called economic significance: is this fact really a big deal or is it small in terms of the big picture of things?”

  • The Sharpe Ratio is a poor evaluation metric because it assumes symmetric returns and ignores tail behavior; strategies with high Sharpe Ratios may carry hidden catastrophic downside risk while low-Sharpe strategies may offer lottery-like positive tails, so the ratio is only valid for roughly normally-distributed returns.

    “it does not take into account the tail behavior... the high Sharpe Ratio strategy, on average it does really well; but then there's a possibility of a catastrophic downside. So, the Sharpe Ratio is only useful if you are evaluating... relatively normally-distributed returns.”

  • A strategy of holding the S&P 500 while writing out-of-the-money calls and puts collects premium that looks like outperformance over calm years, but it caps the upside and magnifies the downside in extreme moves, creating dangerous negative skewness whose extra return is merely compensation for hidden catastrophic risk that simply hadn't materialized.

    “this person has changed the payoffs of the S&P 500, cut off the upside, and magnified the downside. And this extra return that they are getting over the 5 years, well, they are lucky over the 5 years. You haven't seen a big move up or down.”

  • A truly diversified portfolio must be optimized over a third dimension beyond return and volatility, namely skewness: investors dislike negative skew (catastrophic downside tails) and prefer positive skew (lottery-like upside), so negatively skewed assets must offer a higher expected return to compensate.

    “You need to take this third dimension into account, which is, the technical term for it is skewness... people have a preference. They prefer a strategy or a portfolio or an asset return that's got the positive tail.”

  • The Sharpe Ratio is the strategy's excess return divided by its volatility, and it is mathematically linked to the t-statistic, so a high Sharpe Ratio corresponds to a high t-statistic indicating returns significantly greater than zero.

    “there's a direct link, a direct relationship between the Sharpe Ratio and the t-statistic that we were just talking about. So, they are mathematically linked, and a high Sharpe Ratio means a high t-statistic.”

0.53

Survivorship and selection bias manufacture apparent success from pure randomness without skill.

2 pointscentrality 4/5
  • Apparent forecasting skill can be entirely manufactured through survivorship selection: by emailing 100,000 people opposite predictions and only continuing with those who received correct calls, after 10 weeks roughly 97 people will have seen 10 correct picks in a row from a 50-50 strategy with zero skill, illustrating how selection conceals randomness.

    “after that 10th week, there are 97 people that got 10 picks in a row correct. So you harvest 97 suckers out of this... this is basically a strategy of no skill. Basically, 50-50 every single week.”

  • The popular out-of-sample method of holding out recent data is not a reliable fix because researchers already know what happened in the holdout period and can rig variables, can iteratively re-fit until something works, and a genuinely true strategy may still fail out-of-sample by bad luck over a short window.

    “you actually know what's happened in the past... the researcher knows that and might actually stick in some variables in the early part of the data that they know are going to work in the other sample... it is not a panacea to actually go to the other sample method.”

0.44

Finance has different incentive structures that may limit some data mining compared to academia.

5 pointscentrality 2/5
  • Big data is here to stay and findings from it must be evaluated differently to discard flukes, but the real danger is concocting a story or theory only after a finding emerges; sound practice works from first principles about what is reasonable rather than putting the cart before the horse with ex-post rationalizations.

    “people look at the findings and then only after the finding, they concoct a story or a theory, develop ex post some intuition, as to why this should work. And I think that's dangerous. I prefer to work on the basis of first principles: what is reasonable.”

  • Finance has little replication culture because it analyzes shared factual data rather than running human-subject experiments—anyone can rerun a strategy on the same New York Stock Exchange data with one line of code—whereas psychology depends on experiments that genuinely require independent replication.

    “We're not running experiments with human subjects. We're actually looking at data, and the data is a fact... anybody can go in and immediately, with one line of computer code, replicate that.”

  • Psychology sits at the bottom of the scientific hierarchy for publishing non-significant results: over 92% of published psychology papers report a hypothesis confirmed, which incentivizes data mining to find publishable results, a problem compounded by a weak culture of replication.

    “the sort of papers that are published in psychology, over 92% of them are, 'Oh, here's a hypothesis; I did an experiment; and I get support.' So that is a problem. Because it leads to people essentially data mining to find a result”

  • Practitioner investment banks have strong incentives to avoid marketing fluke strategies because a strategy that fails damages reputation, reduces fees and bonuses, so the practitioner community is actually motivated to minimize data mining—unlike academia where editors don't see the pre-submission manipulation and replication is minimal.

    “they actually don't want to market a strategy that turns out to be a fluke. Because that means that it hurts their reputation. It reduces the amount of fees that they get... within the practitioner community, at least, there are strong incentives to reduce the impact of data mining”

  • Moving from two-sigma to five-sigma is not roughly twice as strict; two-sigma implies a 1-in-20 fluke chance while five-sigma implies roughly a 1-in-1.5-million fluke chance, because probabilities fall off non-linearly as you move further into the tail.

    “for a 5-sigma, it's 1 divided by 1.5 million. So, this is a very small”