causal
Output tokens cost more because they are not parallelizable
Output tokens are roughly four times more expensive than input tokens because input (prefill/prompt) can be processed in parallel as a batch, whereas output is autoregressive — each generated token requires re-reading the whole model and KV cache before generating the next, a non-parallel operation.
causalpending
Speaker
Dylan PatelEvidence Quote
“Every time I generate a token, I must not only read the whole entire model into memory and activate it, calculate it to generate the next token, I also have to read the entire KV cache.”
Created: 6/13/2026, 3:31:25 AM
Your notes
Loading notes...