Retries, backoff and jitter

Many failures are brief: a dropped connection, a restarting node, a leader election. Retrying turns them into successes. Retrying carelessly turns a struggling dependency into a dead one, because the retries arrive exactly when it's least able to cope.

#The rules

1. Retry only transient errors: timeouts, connection resets, 503, 429. Never 400, 401, 404 — the request won't get better.
2. Retry only safe operations: idempotent ones, or non-idempotent ones protected by an idempotency key.
3. Back off exponentially: wait base × 2ⁿ after the n-th failure (100 ms, 200 ms, 400 ms…), up to a cap.
4. Add jitter: randomise each wait. Without it, every client that failed at the same instant retries at the same instants, in synchronised waves.
5. Limit attempts and total time, and stay within the caller's deadline (timeouts and deadlines).
6. Honour Retry-After when the server sends it.

#Full jitter

sleep = random_between(0, min(cap, base × 2^attempt))

AWS's analysis of backoff strategies found that "full jitter" spreads load best and finishes work fastest under contention.

#Retry budgets and amplification

Retries multiply across layers. If a client, a gateway, a service and a database driver each make up to 3 attempts, one user action can become 3⁴ = 81 calls to the database. Defences:

• Retry at one layer, usually the one closest to the failure; let other layers fail fast.
• Retry budgets: cap retries to a fraction (say 10%) of normal request volume. When most calls fail, the budget is exhausted and retrying stops — exactly when it would only add load.
• Circuit breakers stop calling a dependency that's clearly down (circuit breakers).

#For queues

Message consumers use the same rules: retry transient failures with increasing delays, and move messages that keep failing to a dead-letter queue rather than retrying forever.