causal

Prompt Recomputation Drives First-Token Delay

The first token is slow because the model must reprocess the entire prompt, and for multi-turn chats the whole message history is recomputed, since storing the key-value state in RAM between turns is more expensive than recomputing it.

causalpending

Speaker

Gavin

Evidence Quote

“It's very expensive to go store all that in RAM while it's waiting for you to write your response. It's usually cheaper just to recompute it.”

Source

Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]— Invest Like the Best
Created: 11/17/2026, 12:35:53 PM

Your notes

Loading notes...