normative
Decode Is A Memory Bandwidth Game at Cluster Level
For decode, performance depends on memory bandwidth across the full scale-up cluster rather than on a single chip, because loading the model and KV cache faster allows more tokens per second per user, so the right question is cluster bandwidth rather than chip bandwidth.
normativepending
Speaker
GavinEvidence Quote
“You should be asking, how much memory bandwidth is on your full scale-up cluster?”
Source
Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]— Invest Like the BestCreated: 11/17/2026, 7:08:01 AM
Your notes
Loading notes...