factual

Lower bound on latency set by weight read time

There is an irreducible lower bound on inference latency for a given hardware configuration: you must read all total parameters from memory into the chips at least once, and even using all available memory bandwidth you cannot go faster than that read time.

factualpending

Speaker

Reiner Pope

Evidence Quote

“I need to read all of my total parameters from memory into the chips, and that takes a certain amount of time. If I use all of my memory bandwidth, I can’t do any better than that.”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...