factual
Pretraining is verifiable via held-out loss
Pretraining progress is extremely easy to verify because loss on a held-out test set is a direct measure, so GPT-7 could be RL-trained to minimize compute needed to reach a fixed loss, and swarms could search for the lowest perplexity.
factualpending
Speaker
Dwarkesh PatelEvidence Quote
“Loss on a held-out test set, which is the basis of pretraining progress, is extremely easy to verify.”
Source
Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development— Dwarkesh Patel PodcastCreated: 11/17/2026, 6:27:26 AM
Your notes
Loading notes...