causal

Output tokens cost more because they are not parallelizable

Output tokens are roughly four times more expensive than input tokens because input (prefill/prompt) can be processed in parallel as a batch, whereas output is autoregressive — each generated token requires re-reading the whole model and KV cache before generating the next, a non-parallel operation.

causalpending

Speaker

Dylan Patel

Evidence Quote

“Every time I generate a token, I must not only read the whole entire model into memory and activate it, calculate it to generate the next token, I also have to read the entire KV cache.”

Source

Deepseek Dylan Patel Nathan Lambert— Lex Fridman Podcast
Created: 6/13/2026, 3:31:25 AM

Your notes

Loading notes...