factual

Pretraining is verifiable via held-out loss

Pretraining progress is extremely easy to verify because loss on a held-out test set is a direct measure, so GPT-7 could be RL-trained to minimize compute needed to reach a fixed loss, and swarms could search for the lowest perplexity.

factualpending

Speaker

Dwarkesh Patel

Evidence Quote

“Loss on a held-out test set, which is the basis of pretraining progress, is extremely easy to verify.”

Source

Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development— Dwarkesh Patel Podcast
Created: 11/17/2026, 6:27:26 AM

Your notes

Loading notes...