time to first token
The delay before an LLM starts producing output (TTFT), a key responsiveness metric for streaming answers.
As defined in Aravind Srinivas
Where it comes up · 2
- Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]3 claims
- Aravind Srinivas1 claim
Most important claims · 4 of 4Search all claims for “time to first token” →
- The first token is slow because the model must reprocess the entire prompt, and for multi-turn chats the whole message history is recomputed, since storing the key-value state in RAM between turns is more expensive than recomputing it.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]
- Following Jeff Dean's tail-latency principle, it's not enough to test a few queries and conclude a product is fast; you must track P90 and P99 latencies because with many servers some queries fail at the tail more often without you realizing it, frustrating users during traffic spikes — so tail latency is tracked at every component (search layer, LLM layer, time-to-first-token, throughput).Aravind Srinivas
- Latency, measured as milliseconds to first token and time between subsequent tokens, is the key bottleneck for real-time AI applications such as conversation, translation, and robotics.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]
- Latency is measured as the delay before the first token returns after a user submits a prompt, and the gap between subsequent tokens, with the first-token delay being much longer.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]