causal

Memory matters most for reasoning inference

For reasoning-model inference, memory (bandwidth and capacity) is the most important of the three GPU axes (FLOPS, memory, interconnect), because the KV cache grows quadratically with context length and reasoning models generate tens of thousands of tokens, creating memory pressure that limits how many users can be served simultaneously.

causalpending

Speaker

Nathan Lambert

Evidence Quote

“your memory usage is going way up with these reasoning models and you still have a lot of users, so effectively the cost to serve multiplies by a ton.”

Source

Deepseek Dylan Patel Nathan Lambert— Lex Fridman Podcast
Created: 6/13/2026, 3:31:25 AM

Your notes

Loading notes...