factual

Optimal batch size is about 300 times sparsity

Setting weight-fetch time equal to weight-multiply time yields a hardware constant (FLOPs/byte ratio) of roughly 300 across recent GPUs, so the optimal batch size is approximately 300 times the model's sparsity ratio—e.g., DeepSeek activating 32 of 256 experts gives a sparsity of 8, implying a batch around 2,000-3,000 tokens; this depends only on sparsity, not model scale.

factualpending

Speaker

Reiner Pope

Evidence Quote

“batch size needs to be bigger than approximately 300 times sparsity”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...