factual

Optimal batch is tiny one-thousandth of Gemini traffic

Converting batch size to throughput (batch times ~60-64 per second) gives roughly 128,000 tokens per second for an optimal-batch system, which is only about one-thousandth of Gemini's reported hundreds of millions of tokens per second worldwide—implying batching provides only a modest push toward centralization.

factualpending

Speaker

Reiner Pope

Evidence Quote

“This ends up being around 2,000 x 64, so 128,000 tokens per second... The numbers I remember from some announcements of Gemini last year were in the hundreds of millions of tokens per second worldwide. This is one-thousandth of that.”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...