factual

Pretraining data gains are ordinary automatable ML work

Pretraining data improvements are largely normal ML research, basically figuring out which filtering or synthetic data techniques lower loss on a held-out corpus, and open corpora grew from roughly 9B tokens in OpenWebText to far larger, classifier-filtered scrapes like UltraFineWeb.

factualpending

Speaker

Dwarkesh Patel

Evidence Quote

“these “data” improvements in pretraining are very much in line with normal AI research work”

Source

Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development— Dwarkesh Patel Podcast
Created: 11/17/2026, 6:27:26 AM

Your notes

Loading notes...