data mining
Searching through many variables until a statistically significant pattern appears by chance, producing relationships that don't hold out of sample.
As defined in Cliff Asness
Other definitions
The practice of searching through large datasets for patterns or relationships without theoretical justification, risking the discovery of spurious correlations that don't hold out-of-sample.
Volatility Laundering in PE, International Diversification, Value Vs Growth and Systematic Investing
Where it comes up · 13
- Campbell Harvey on Randomness, Skill, and Investment Strategies10 claims
- RARE Alex Karp Interview: Palantir Philosophy Explained! (2009)4 claims
- Volatility Laundering in PE, International Diversification, Value Vs Growth and Systematic Investing3 claims
- Cliff Asness: A Brief and Biased Survey of Quantitative Investing3 claims
- Ian Ayres on Super Crunchers and the Power of Data 10/22/20072 claims
- The Past, The Present & Future of Quant2 claims
- Compound Your Face Off2 claims
- Susan Athey on Machine Learning, Big Data, and Causation2 claims
and 5 more
Most important claims · 10 of 33Search all claims for “data mining” →
- The first person to publish a discovery often suffers a 'winner's curse': because of the data mining that preceded publication, the reported effect is likely overstated or nonexistent, so replications typically find a much smaller or absent effect.Campbell Harvey on Randomness, Skill, and Investment Strategies
- Spurious findings arise not just from the number of tests but from the full set of discretionary researcher choices—sample start date, trimming outliers, choice of estimation method—and because journal editors and investors never see these hidden choices, the problem is a serious and largely invisible source of false results.Campbell Harvey on Randomness, Skill, and Investment Strategies
- The popular out-of-sample method of holding out recent data is not a reliable fix because researchers already know what happened in the holdout period and can rig variables, can iteratively re-fit until something works, and a genuinely true strategy may still fail out-of-sample by bad luck over a short window.Campbell Harvey on Randomness, Skill, and Investment Strategies
- Big data is here to stay and findings from it must be evaluated differently to discard flukes, but the real danger is concocting a story or theory only after a finding emerges; sound practice works from first principles about what is reasonable rather than putting the cart before the horse with ex-post rationalizations.Campbell Harvey on Randomness, Skill, and Investment Strategies
- Every empirical regularity that predictably works (or fails) does so for one of three reasons: complete accident (data mining), a behavioral story (someone making a systematic error you exploit), or risk (in a rational world you get paid for bearing risk); these are not mutually exclusive and can coexist.Cliff Asness
- Repeatedly testing different variants until one shows statistical significance, such as trying many shirt colors to find one that predicts coin flips, produces spurious findings that look causal, and this can happen unknowingly in investing when many theories and parameters are tested, and computers make searching for patterns much easier.The Search for the Truth with the Anonymous Master
- If you measure a thousand characteristics on a thousand patients and look for what the 15 best-responding patients share, you will almost certainly find common traits by chance alone, letting you 'prove' a spurious subgroup treatment effect even when the drug has no real effect; this is why the FDA requires pre-analysis plans.Susan Athey on Machine Learning, Big Data, and Causation
- A simple value strategy with a three Sharpe ratio would have already been arbitraged away, so the real persistent strategies are those that pass tests across many places and time periods with a good story, while data-mined ones that look excellent will not repeat.The Past, The Present & Future of Quant
- A circulating paper sorting stock performance by the first letter of the company name found one of 26 letters to be statistically significant, which is exactly what the jellybean multiple-testing problem predicts and demonstrates how researchers fail to adjust for data mining.Campbell Harvey on Randomness, Skill, and Investment Strategies
- A researcher at a top investment bank presented the 17th monthly lag of US Industrial Production as a significant market predictor, but admitted it was 'the only one that worked' out of roughly 24 lag tests, so adjusting significance for the number of tests reveals it as insignificant—a finance version of the jellybean problem.Campbell Harvey on Randomness, Skill, and Investment Strategies