factual
Inference time is max of compute and memory time
The time to run an LLM inference can be lower-bounded as the maximum of two terms: the compute time (batch size times active parameters divided by chip FLOPs) and the memory time (total parameters plus KV cache bytes, divided by memory bandwidth); attention compute is small enough to ignore, giving strong predictive power from a simple model.
factualpending
Speaker
Reiner PopeEvidence Quote
“We’re going to consider two different aspects: the time it takes to do the memory fetches, and the time it takes to do the compute. It will turn out that this gives us very strong predictive power, even with a simple model.”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...