normative
Track tail latency, not average latency
Following Jeff Dean's tail-latency principle, it's not enough to test a few queries and conclude a product is fast; you must track P90 and P99 latencies because with many servers some queries fail at the tail more often without you realizing it, frustrating users during traffic spikes — so tail latency is tracked at every component (search layer, LLM layer, time-to-first-token, throughput).
normativepending
Speaker
Aravind SrinivasEvidence Quote
“we track it at every single component of our system, be it the search layer or the LLM layer.”
Created: 6/13/2026, 3:26:05 AM
Your notes
Loading notes...