Tail latency

Tail latency is the latency of the slowest requests: the 99th percentile (p99) or 99.9th (p999). Averages hide it, and at scale it's what users actually notice.

#Why the tail matters

• Heavy users see it most. Someone who loads 100 pages a day hits the p99 about once a day, and those users are often your best customers.
• Fan-out amplifies it. If a page needs answers from 50 backend calls in parallel and each has a 1% chance of being slow, the chance that at least one is slow is:

1 − 0.99^50 ≈ 0.39  → 39% of page loads see a p99-slow call

The more a request fans out, the more the backend's tail becomes the user's median.

#Where tail latency comes from

Garbage-collection pauses, cache misses, a noisy neighbour on a shared host, queueing behind a large request, a compaction or vacuum running, a retransmitted TCP packet, a cold connection needing a TLS handshake. Most are brief and random, which is why the techniques below work.

#Reducing it

• Hedged requests. Send the request to one replica; if no answer arrives within the p95 latency, send it to a second and use whichever answers first. It adds a few percent of load and can cut p99 dramatically. Only do this for idempotent reads.
• Tighter timeouts with fallbacks for optional calls, so one slow dependency doesn't hold the page (timeouts and deadlines).
• Reduce fan-out by batching calls or precomputing results.
• Keep queues short and separate large requests from small ones, so small ones don't wait behind big ones.
• Measure correctly. Record latency in histograms and compute percentiles over all requests. Averaging p99 values from several hosts doesn't give the fleet's p99.