out-of-sample test
A test of an investment or statistical finding using data that was not used to develop or optimize the original strategy, used to distinguish real patterns from random data mining artifacts.
As defined in Volatility Laundering in PE, International Diversification, Value Vs Growth and Systematic Investing
Other definitions
A validation method for investment strategies where a rule discovered on one dataset is tested on independent data (different time period, different market, different asset class) to verify the pattern is real rather than a statistical artifact
Cliff Asness: A Brief and Biased Survey of Quantitative Investing
Where it comes up · 3
- The Past, The Present & Future of Quant4 claims
- Cliff Asness: A Brief and Biased Survey of Quantitative Investing3 claims
- Volatility Laundering in PE, International Diversification, Value Vs Growth and Systematic Investing2 claims
Most important claims · 9 of 9Search all claims for “out-of-sample test” →
- Momentum effects also passed international out-of-sample tests, which is important because if a behavioral pattern only works in the US due to specific psychological reasons, it strains credibility; if it works globally, it's more likely to be a fundamental market inefficiency.Volatility Laundering in PE, International Diversification, Value Vs Growth and Systematic Investing
- To distinguish random data mining results from real effects, researchers must verify: (1) a plausible theory explaining why the result should exist, and (2) out-of-sample tests showing the result holds in data the researcher had not previously examined.Volatility Laundering in PE, International Diversification, Value Vs Growth and Systematic Investing
- Economics cannot use the scientific method's randomized A/B experiments because the world is observed only once, so each event like Pandora's box cannot be unrung, and plausible economic logic plus out-of-sample tests must substitute for controlled trials.The Past, The Present & Future of Quant
- Asness tested value and momentum strategies across multiple countries and markets beyond the US, providing geographical out-of-sample tests that validated the ideas work across different regions, reducing concern that findings were specific to US market conditions.Cliff Asness: A Brief and Biased Survey of Quantitative Investing
- Testing CAPE timing by bucketing returns by Shiller P/E over the full period is cheating because it uses future information, so a fair test must look only backwards on rolling windows, and under that test CAPE still shows a positive but narrow edge.The Past, The Present & Future of Quant
- Testing an academic author's factor story across the asset classes, geographies, and time periods where it should apply raises the hurdle for data mining, since a finding that holds in similar spirit across contexts is less likely to be spurious.The Past, The Present & Future of Quant
- The Shiller CAPE predicts 10-year equity returns and should lower expected returns when high (e.g. 35 vs an average of 16), but using it to time the market out of sample yields only a narrow, weak edge across about 120-140 years and is not a career-making strategy.The Past, The Present & Future of Quant
- Managed Futures (systematic trend-following strategies) have worked as a strategy for over a hundred years—they buy what's been going up and sell what's been going down—which is a long out-of-sample test validating momentum as a real phenomenon.Cliff Asness: A Brief and Biased Survey of Quantitative Investing
- In the 1990s at Goldman Sachs, Asness was asked to implement quantitative models on real money, which served as an out-of-sample test of whether the academic research on value and momentum would actually work in practice with real capital.Cliff Asness: A Brief and Biased Survey of Quantitative Investing