factual
The data wall is an underrated threat to scaling
The data wall is underrated and could cause stagnation: Common Crawl is about 30 trillion tokens and Llama-3 was already trained on 15 trillion—near the limit; repeating data only helps to a point (a Boaz Barak scaling-laws paper finds returns go to zero after about 16 repetitions), so at most ~10x more effective data is available, giving only ~100x effective compute over GPT-4, roughly two more years of progress—meaning we'll learn by 2025-2026 whether the data wall can be cracked.
factualpending
Speaker
Leopold AschenbrennerEvidence Quote
“Llama-3 is already at the limit of data. Maybe we can get 10x more by repeating data. At most that's a 100x model than GPT-4, a 100x effective compute from GPT-4. That's not that much.”
Source
Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:30:02 AM
Your notes
Loading notes...