factual
Models may be 100x over-trained beyond Chinchilla
Equalizing pre-training, RL, and inference compute implies the number of inference tokens roughly equals the number of pre-training tokens; estimating ~50M tokens/sec for two months gives ~200 trillion inference tokens against a Chinchilla-optimal ~2 trillion for a ~100B-active-parameter model, suggesting frontier models are roughly 100x over-trained relative to Chinchilla-optimal.
factualpending
Speaker
Reiner PopeEvidence Quote
“The ratio of this two hundred trillion or a hundred trillion parameters over the Chinchilla optimal of two trillion, that’s the amount it’s over-trained. Which is a factor of a hundred over-trained.”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...