memory bandwidth
The rate at which data can be read from or written to a GPU's memory; especially important for serving reasoning models with large KV caches.
As defined in Deepseek Dylan Patel Nathan Lambert
Where it comes up · 3
- Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served9 claims
- Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]1 claim
- Deepseek Dylan Patel Nathan Lambert1 claim
Most important claims · 10 of 11Search all claims for “memory bandwidth” →
- Inference is memory-bound because each parameter must be loaded from memory and used once, so batching many users to reuse each loaded weight amortizes the expensive load, and a transformer ASIC can batch around 2,500 users versus about 64 on GPUs.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]
- Setting weight-fetch time equal to weight-multiply time yields a hardware constant (FLOPs/byte ratio) of roughly 300 across recent GPUs, so the optimal batch size is approximately 300 times the model's sparsity ratio—e.g., DeepSeek activating 32 of 256 experts gives a sparsity of 8, implying a batch around 2,000-3,000 tokens; this depends only on sparsity, not model scale.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Larger scale-up domains massively reduce weight memory-fetch latency because all GPUs in a scale-up domain load weights in parallel, so effective bandwidth equals scale-up size times per-GPU bandwidth; this term grew ~8x from Hopper while per-GPU bandwidth only grows ~1.5-2x per generation, meaning scale-up size—not memory capacity—solves the bandwidth/latency problem.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- The ~20 millisecond batch interval comes from wanting to read all of HBM capacity into the chip exactly once per forward pass—capacity divided by bandwidth—which is about 20 milliseconds across many HBM generations (e.g., Rubin's ~288 GB over ~20 TB/s ≈ 15 ms); running slower would wastefully re-read read-only weights and KVs.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Output (decode) being charged roughly 3-5x more than input (prefill) indicates that prefill is compute-limited while decode is memory-bandwidth-limited: processing many tokens in one pass amortizes the weight memory fetch across the pass length, so the per-token memory cost falls with longer passes, revealing the labs are heavily memory-bandwidth bottlenecked during decode.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- From a pure bandwidth-time perspective, increasing sparsity is a win as long as you have enough users to fill larger batches, because the doubled weight-memory fetches are amortized away by running a bigger batch; the binding cost is instead the additional memory capacity the larger total parameter count consumes.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- For reasoning-model inference, memory (bandwidth and capacity) is the most important of the three GPU axes (FLOPS, memory, interconnect), because the KV cache grows quadratically with context length and reasoning models generate tens of thousands of tokens, creating memory pressure that limits how many users can be served simultaneously.Deepseek Dylan Patel Nathan Lambert
- If, as Dario claims, in-context learning rather than continual learning suffices for AGI, then matching a month of working with an employee requires roughly hundred-million-token context lengths, which would demand a major ML-infra change to memory bandwidth (unless sparse attention substitutes)—a constraint Pope sees no good path to solving given HBM stagnation.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- There is an irreducible lower bound on inference latency for a given hardware configuration: you must read all total parameters from memory into the chips at least once, and even using all available memory bandwidth you cannot go faster than that read time.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- The compute cost of long context (including the quadratic/linear attention term) has such a low slope that it is well-approximated as flat until millions of tokens; the binding constraints on very long context are memory bandwidth and memory capacity, and context lengths have plateaued around 100-200K because there is no good solution to this memory wall—sparse attention helps via its square-root term but going too sparse degrades quality.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served