# Retries, backoff and jitter

> Retry only what is safe, wait longer each time, add randomness, and cap the total.

Concept · Reliability · https://levelcode.org/concepts/retries-with-backoff

Many failures are brief: a dropped connection, a restarting node, a leader election. Retrying turns them into successes. Retrying carelessly turns a struggling dependency into a dead one, because the retries arrive exactly when it's least able to cope.

## The rules

1. **Retry only transient errors:** timeouts, connection resets, `503`, `429`. Never `400`, `401`, `404` — the request won't get better.
2. **Retry only safe operations:** idempotent ones, or non-idempotent ones protected by an [idempotency key](https://levelcode.org/concepts/idempotency-keys).
3. **Back off exponentially:** wait base × 2ⁿ after the n-th failure (100 ms, 200 ms, 400 ms…), up to a cap.
4. **Add jitter:** randomise each wait. Without it, every client that failed at the same instant retries at the same instants, in synchronised waves.
5. **Limit attempts and total time,** and stay within the caller's deadline ([timeouts and deadlines](https://levelcode.org/concepts/timeouts)).
6. **Honour `Retry-After`** when the server sends it.

## Full jitter

```text
sleep = random_between(0, min(cap, base × 2^attempt))
```

AWS's analysis of backoff strategies found that "full jitter" spreads load best and finishes work fastest under contention.

## Retry budgets and amplification

Retries multiply across layers. If a client, a gateway, a service and a database driver each make up to 3 attempts, one user action can become 3⁴ = 81 calls to the database. Defences:

- **Retry at one layer**, usually the one closest to the failure; let other layers fail fast.
- **Retry budgets:** cap retries to a fraction (say 10%) of normal request volume. When most calls fail, the budget is exhausted and retrying stops — exactly when it would only add load.
- **Circuit breakers** stop calling a dependency that's clearly down ([circuit breakers](https://levelcode.org/concepts/circuit-breakers)).

## For queues

Message consumers use the same rules: retry transient failures with increasing delays, and move messages that keep failing to a [dead-letter queue](https://levelcode.org/concepts/dead-letter-queues) rather than retrying forever.