forecast

Pre-training will run out of finite data

Pre-training will run out of data because data is very clearly finite, forcing a choice between a souped-up new pre-training recipe, RL, or something else; meanwhile RL now consumes more compute than pre-training because long rollouts cost a lot of compute for relatively little learning per rollout.

forecastpending

Speaker

Ilya Sutskever

Evidence Quote

“pre-training will run out of data. The data is very clearly finite.”

Source

Ilya Sutskever – We're moving from the age of scaling to the age of research— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:24:31 AM

Your notes

Loading notes...