factual

Inference time is max of compute and memory time

The time to run an LLM inference can be lower-bounded as the maximum of two terms: the compute time (batch size times active parameters divided by chip FLOPs) and the memory time (total parameters plus KV cache bytes, divided by memory bandwidth); attention compute is small enough to ignore, giving strong predictive power from a simple model.

factualpending

Speaker

Reiner Pope

Evidence Quote

“We’re going to consider two different aspects: the time it takes to do the memory fetches, and the time it takes to do the compute. It will turn out that this gives us very strong predictive power, even with a simple model.”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...