factual
Optimal batch is tiny one-thousandth of Gemini traffic
Converting batch size to throughput (batch times ~60-64 per second) gives roughly 128,000 tokens per second for an optimal-batch system, which is only about one-thousandth of Gemini's reported hundreds of millions of tokens per second worldwide—implying batching provides only a modest push toward centralization.
factualpending
Speaker
Reiner PopeEvidence Quote
“This ends up being around 2,000 x 64, so 128,000 tokens per second... The numbers I remember from some announcements of Gemini last year were in the hundreds of millions of tokens per second worldwide. This is one-thousandth of that.”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...