Pre-training appears to be reaching data saturation, requiring exploration of alternative approaches like modified pre-training recipes or reinforcement learning paradigms to continue generating progress.
forecastpending
Speaker
Ilya SutskeverEvidence Quote
“At some point though, pre-training will run out of data. The data is very clearly finite.”
Created: 8/12/2026, 6:07:19 PM
Your notes
Loading notes...