forecast
Pre-training will run out of finite data
Pre-training will run out of data because data is very clearly finite, forcing a choice between a souped-up new pre-training recipe, RL, or something else; meanwhile RL now consumes more compute than pre-training because long rollouts cost a lot of compute for relatively little learning per rollout.
forecastpending
Speaker
Ilya SutskeverEvidence Quote
“pre-training will run out of data. The data is very clearly finite.”
Source
Ilya Sutskever – We're moving from the age of scaling to the age of research— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:24:31 AM
Your notes
Loading notes...