factual
Lower bound on latency set by weight read time
There is an irreducible lower bound on inference latency for a given hardware configuration: you must read all total parameters from memory into the chips at least once, and even using all available memory bandwidth you cannot go faster than that read time.
factualpending
Speaker
Reiner PopeEvidence Quote
“I need to read all of my total parameters from memory into the chips, and that takes a certain amount of time. If I use all of my memory bandwidth, I can’t do any better than that.”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...