data wall
The looming limit on training data—the world's stock of high-quality text (e.g., Common Crawl) is nearly exhausted, threatening to halt the gains from scaling pre-training unless a new paradigm is found.
As defined in Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history
Other definitions
A theoretical ceiling on AI model improvement caused by scarcity of training data—the finite amount of existing text, images, and other data in the world that can be used to train models
Bret Taylor: Facebook, Salesforce, OpenAI, and Sierra
Where it comes up · 4
- Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history3 claims
- Vlad Tenev - Navigating Robinhood's Evolution - [Invest Like the Best, EP.384]2 claims
- Bret Taylor: Facebook, Salesforce, OpenAI, and Sierra2 claims
- Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history2 claims
Most important claims · 9 of 9Search all claims for “data wall” →
- People tend to underestimate the rate of change and overestimate the difficulty of being in the trenches of a field. Being in close focus on immediate technical challenges (debugging current models, fixing data issues) reduces visibility into macro trends (exponential capability scaling, approaching AGI) and makes algorithmic progress seem less impressive than it actually is.Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history
- Models have already trained on nearly all public internet data, creating a data wall; synthetic data generated by the previous model generation will probably be much less valuable than human data, because otherwise it would be a perpetual motion machine for capability gains.Vlad Tenev - Navigating Robinhood's Evolution - [Invest Like the Best, EP.384]
- The primary inputs limiting AGI progress are data (textual training data is 'waning' with a potential 'data wall'), compute (capital-intensive, with hardware costs that may not be recovered), and algorithms (with potential plateaus in scaling laws even after breakthroughs like reasoning models).Bret Taylor: Facebook, Salesforce, OpenAI, and Sierra
- Reasoning models represent a breakthrough in generating net new ideas and synthetic data quality because they're not just regurgitating training data, which could help overcome the 'data wall' facing traditional scaling approaches.Bret Taylor: Facebook, Salesforce, OpenAI, and Sierra
- People overrate compute and underrate the algorithmic secrets: the default ~0.5 OOM/year of algorithmic progress means a few years of protected lead compounds to a 10-100x effective compute advantage; the critical secret is the new paradigm to get past the data wall ('AlphaGo step two' self-play RL)—if China can't steal that, they're stuck, but if they can, they're off to the races.Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history
- Large-scale engineering to make training runs work is tacit knowledge that China will eventually figure out, so it's not a durable source of advantage, but self-play RL for overcoming the data wall is more novel.Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history
- The data wall is underrated and could cause stagnation: Common Crawl is about 30 trillion tokens and Llama-3 was already trained on 15 trillion—near the limit; repeating data only helps to a point (a Boaz Barak scaling-laws paper finds returns go to zero after about 16 repetitions), so at most ~10x more effective data is available, giving only ~100x effective compute over GPT-4, roughly two more years of progress—meaning we'll learn by 2025-2026 whether the data wall can be cracked.Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history
- Learning has two phases: pre-training is like passive lecturing where words fly by, while real learning happens by reading, thinking, having an internal monologue, attempting practice problems, and failing until it clicks—humans transition from being pre-trained (grade school) to teaching themselves (by college), and models are just now entering that self-learning regime via self-play, synthetic data, and RL.Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history
- Frontier models such as GPT-4 and Claude 3.5 were trained on nearly all public internet data, creating a data wall where further progress requires new data sources.Vlad Tenev - Navigating Robinhood's Evolution - [Invest Like the Best, EP.384]