normative

Decode Is A Memory Bandwidth Game at Cluster Level

For decode, performance depends on memory bandwidth across the full scale-up cluster rather than on a single chip, because loading the model and KV cache faster allows more tokens per second per user, so the right question is cluster bandwidth rather than chip bandwidth.

normativepending

Speaker

Gavin

Evidence Quote

“You should be asking, how much memory bandwidth is on your full scale-up cluster?”

Source

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]— Invest Like the Best
Created: 11/17/2026, 7:08:01 AM

Your notes

Loading notes...