Pre-training appears to be reaching data saturation, requiring exploration of alternative approaches like modified pre-training recipes or reinforcement learning paradigms to continue generating progress.

forecastpending

Speaker

Ilya Sutskever

Evidence Quote

“At some point though, pre-training will run out of data. The data is very clearly finite.”

Source

Ilya Sutskever – We're moving from the age of scaling to the age of research— Dwarkesh Patel
Created: 8/12/2026, 6:07:19 PM

Your notes

Loading notes...