normative

First-principles deduction of pre-training data from token throughput

Because optimal design equalizes inference and pre-training tokens, if a lab reveals tokens served per second you can back out roughly how much data went into pre-training; even with 50% error this is empowering, because you can first-principles these numbers by setting cost A equal to cost B.

normativepending

Speaker

Reiner Pope

Evidence Quote

“This is why you should just approximate everywhere, because there are big error bars on this. But it’s kind of empowering to just set A equal to B and figure it out.”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...