causal

Pipelining cannot shard the KV cache footprint

Pipelining cannot reduce per-GPU KV cache memory: while fewer layers per rack means storing fewer layers of KVs, keeping all racks busy requires proportionally more sequences in flight, so the number of micro-batches equals the number of pipeline stages and the pipeline-stage factor exactly cancels—the weight footprint shrinks with pipelining but the activation/KV footprint stays constant.

causalpending

Speaker

Reiner Pope

Evidence Quote

“if you increase the number of pipeline stages, the memory footprint for the number of weights keeps going down and down and down, but the memory footprint for the number of activations stays constant. So it doesn’t actually work.”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...