causal

20ms train interval from HBM drain time

The ~20 millisecond batch interval comes from wanting to read all of HBM capacity into the chip exactly once per forward pass—capacity divided by bandwidth—which is about 20 milliseconds across many HBM generations (e.g., Rubin's ~288 GB over ~20 TB/s ≈ 15 ms); running slower would wastefully re-read read-only weights and KVs.

causalpending

Speaker

Reiner Pope

Evidence Quote

“we want to read all of the memory capacity into the chip. That is capacity divided by bandwidth. That tends to be 20 milliseconds on many different generations of HBM.”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...