normative

Track tail latency, not average latency

Following Jeff Dean's tail-latency principle, it's not enough to test a few queries and conclude a product is fast; you must track P90 and P99 latencies because with many servers some queries fail at the tail more often without you realizing it, frustrating users during traffic spikes — so tail latency is tracked at every component (search layer, LLM layer, time-to-first-token, throughput).

normativepending

Speaker

Aravind Srinivas

Evidence Quote

“we track it at every single component of our system, be it the search layer or the LLM layer.”

Source

Aravind Srinivas— Lex Fridman Podcast
Created: 6/13/2026, 3:26:05 AM

Your notes

Loading notes...