normative
First-principles deduction of pre-training data from token throughput
Because optimal design equalizes inference and pre-training tokens, if a lab reveals tokens served per second you can back out roughly how much data went into pre-training; even with 50% error this is empowering, because you can first-principles these numbers by setting cost A equal to cost B.
normativepending
Speaker
Reiner PopeEvidence Quote
“This is why you should just approximate everywhere, because there are big error bars on this. But it’s kind of empowering to just set A equal to B and figure it out.”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...