# Designing for failure

> Stop one slow dependency from taking everything down: timeouts and deadlines, safe retries with backoff and jitter, circuit breakers, bulkheads, load shedding and graceful degradation.

System Design roadmap · Communication & reliability · 21 min read · https://levelcode.org/roadmaps/system-design/designing-for-failure

In a system of many services, something is always failing: a node is restarting, a disk is slow, a dependency is returning errors, a network link is dropping packets. That's normal, and users shouldn't notice. What turns normal failure into an outage is **propagation** — one slow component ties up the threads of its callers, whose callers then stall, until the whole system is waiting on one sick database. This lesson covers the patterns that contain failure, in the order requests meet them.

## How a cascading failure happens

A typical sequence:

1. The recommendations service's database gets slow: p99 goes from 20 ms to 4 seconds.
2. The product-page service calls recommendations with no timeout. Each request now holds a worker for 4 seconds instead of 50 ms.
3. By [Little's law](https://levelcode.org/concepts/littles-law), in-flight requests grow 80-fold. The product-page service runs out of workers and connections.
4. Product pages time out at the load balancer. Clients retry. Load doubles.
5. The checkout service shares the product-page service's connection pool for price lookups. Checkout stops working.

The database problem was in a non-essential feature. The outage was in checkout. Every pattern below breaks one link in that chain.

## Timeouts and deadlines

**Every network call needs a timeout.** Without one, a hung dependency holds resources forever. See [timeouts and deadlines](https://levelcode.org/concepts/timeouts).

Set timeouts from data, not instinct: a little above the dependency's p99.9 latency under normal load, and well below your own caller's patience. If a dependency normally answers in 30 ms at p99.9, a 200 ms timeout leaves room for noise without letting a stall last 30 seconds. Distinguish the **connect** timeout (short: a few hundred milliseconds) from the **request** timeout.

Better still, propagate a **deadline**. If the user's request has a 2-second budget and 1.5 seconds are gone, the next call should get at most 0.5 seconds — there's no point working on an answer nobody will wait for. Go's `context` does this naturally:

```go
func (s *Server) handleProduct(w http.ResponseWriter, r *http.Request) {
	// The whole request gets 800 ms; every call below inherits what's left.
	ctx, cancel := context.WithTimeout(r.Context(), 800*time.Millisecond)
	defer cancel()

	product, err := s.catalog.Get(ctx, r.PathValue("id")) // essential
	if err != nil {
		http.Error(w, "unavailable", http.StatusServiceUnavailable)
		return
	}

	// Non-essential: a tighter budget of its own, and a fallback on failure.
	recCtx, recCancel := context.WithTimeout(ctx, 150*time.Millisecond)
	defer recCancel()
	recs, err := s.recs.For(recCtx, product.ID)
	if err != nil {
		recs = s.popularFallback() // degrade, don't fail
	}
	render(w, product, recs)
}
```

gRPC sends the deadline to the server in a header, so a downstream service can give up when its caller already has. For plain HTTP, pass the remaining budget in a header and apply it on the receiving side.

## Retries, without the stampede

Many failures are brief: a dropped connection, a node restarting, a leader election. A retry turns those into successes. Done carelessly, retries turn a struggling service into a dead one. See [retries, backoff and jitter](https://levelcode.org/concepts/retries-with-backoff).

**Retry only what's safe and might succeed.** Retry timeouts, connection errors, 503 and 429. Don't retry 400 or 404 — the request is wrong and will stay wrong. Retry non-idempotent operations (creating an order, charging a card) only with an [idempotency key](https://levelcode.org/concepts/idempotency-keys) the server uses to deduplicate.

**Back off exponentially, with jitter.** Wait longer after each attempt, and randomise the wait so thousands of clients that failed at the same moment don't all retry at the same moment. "Full jitter" — a random wait between zero and the exponential cap — spreads retries best:

```go
// retry calls fn up to attempts times, sleeping a random duration in
// [0, min(maxDelay, base·2^n)) between tries, and stops when ctx is done.
func retry(ctx context.Context, attempts int, base, maxDelay time.Duration, fn func(context.Context) error) error {
	var err error
	for n := 0; n < attempts; n++ {
		if err = fn(ctx); err == nil || !retryable(err) {
			return err
		}
		if n == attempts-1 {
			break
		}
		backoff := min(maxDelay, base<<n)
		select {
		case <-time.After(rand.N(backoff)):
		case <-ctx.Done():
			return ctx.Err()
		}
	}
	return err
}
```

**Cap retries at one layer.** If the browser, the gateway, the API and the database client each make three attempts, one user action can produce 3 × 3 × 3 × 3 = 81 calls to the database while it's already overloaded. Retry at one layer — usually closest to the failure — and pass errors up from the others.

**Use a retry budget.** Allow retries to add at most, say, 10% to normal traffic. When most calls are failing, a retry is more likely to add load than to succeed, so the budget runs out and you stop retrying. Several gRPC and service-mesh implementations offer retry budgets built in.

## Circuit breakers

A timeout bounds how long *one* call waits. When a dependency is down, though, every call still waits the full timeout and then fails. A **circuit breaker** notices the pattern and stops calling for a while. See [circuit breakers](https://levelcode.org/concepts/circuit-breakers).

It has three states:

- **Closed** — normal. Calls pass through; failures are counted over a sliding window.
- **Open** — the failure rate crossed a threshold (for example 50% of at least 20 calls in 10 seconds). Calls fail **immediately** without touching the dependency, for a cool-down period. The dependency gets room to recover, and callers stop waiting.
- **Half-open** — after the cool-down, a few trial calls are let through. If they succeed, close; if not, open again.

```sequence
title: A circuit breaker opens and recovers
participants: Caller, Breaker, Dependency
Caller -> Breaker: call
Breaker -> Dependency: request
Dependency --> Breaker: timeout (failure 12 of 20)
> The failure rate crosses 50%. The breaker opens.
Caller -> Breaker: call
Breaker --> Caller: fail fast (open), use fallback
> No request reaches the dependency for 30 seconds.
Caller -> Breaker: call (cool-down over)
Breaker -> Dependency: trial request (half-open)
Dependency --> Breaker: 200 OK
Breaker --> Caller: OK; breaker closes
```

Breakers pair with fallbacks: while open, return cached data, a default, or a reduced response. Libraries exist for every language (for example `sony/gobreaker` in Go), and service meshes such as Envoy implement the same idea as outlier detection.

## Bulkheads

Ships are divided into watertight compartments so one breach doesn't sink them. A **bulkhead** in software gives each dependency its own limited pool of resources, so a slow one can exhaust only its own share. See [bulkheads](https://levelcode.org/concepts/bulkheads).

In the cascading failure above, checkout died because it shared a pool with a feature that hung. With separate limits — at most 20 concurrent calls to recommendations, its own connection pool for prices — recommendations could only ever tie up 20 workers.

```go
// A bulkhead: at most n concurrent calls; extra callers fail fast instead of queueing.
type Bulkhead struct{ slots chan struct{} }

func NewBulkhead(n int) *Bulkhead { return &Bulkhead{slots: make(chan struct{}, n)} }

var ErrBulkheadFull = errors.New("bulkhead full")

func (b *Bulkhead) Do(ctx context.Context, fn func(context.Context) error) error {
	select {
	case b.slots <- struct{}{}:
		defer func() { <-b.slots }()
		return fn(ctx)
	default:
		return ErrBulkheadFull
	}
}
```

The same idea applies at every scale: separate thread pools per dependency, separate database connection pools for user traffic and batch jobs, separate clusters for internal tools and customer traffic, and cell-based architectures where each "cell" serves a slice of customers so a bad deploy or poison request affects only that slice.

## Load shedding and backpressure

When demand exceeds capacity, something won't be served. You choose whether that's *some requests, quickly rejected*, or *every request, slowly timing out*. **Load shedding** chooses the former. See [load shedding](https://levelcode.org/concepts/load-shedding).

- **Reject early and cheaply.** Check before doing any work: if in-flight requests or queue wait time exceed a limit, return `503` with `Retry-After`. A rejection costs microseconds; a request that times out after 10 seconds of work wasted all of it.
- **Shed by priority.** Drop background and non-interactive traffic first, then anonymous traffic, and keep checkout and login last.
- **Bound every queue.** An unbounded queue under overload grows until requests wait longer than anyone will — every one then times out, and the server does work nobody receives. A bounded queue plus rejection is [backpressure](https://levelcode.org/concepts/backpressure): it pushes the overload signal back to callers, who can slow down.

```go
// shed rejects requests once more than limit are in flight, before any work is done.
func shed(limit int64, next http.Handler) http.Handler {
	var inflight atomic.Int64
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if inflight.Add(1) > limit {
			inflight.Add(-1)
			w.Header().Set("Retry-After", "1")
			http.Error(w, "overloaded", http.StatusServiceUnavailable)
			return
		}
		defer inflight.Add(-1)
		next.ServeHTTP(w, r)
	})
}
```

Rate limiting (previous lesson) caps each *client*. Load shedding caps the *server*, whatever the mix of clients. You want both.

## Graceful degradation

Decide in advance what the product does when each dependency is gone, and build it. See [graceful degradation](https://levelcode.org/concepts/graceful-degradation).

| Dependency down | Degraded behaviour |
|---|---|
| Recommendations | Show popular items, or nothing |
| Search | Show categories and recent items; tell users search is unavailable |
| Reviews | Hide the section |
| Payment provider A | Fail over to provider B, or queue the order for retry |
| Personalisation | Serve the generic page from the CDN |

Classify features as **essential** (checkout, login) and **optional** (everything that decorates the page). Optional features get short timeouts, breakers and fallbacks; they must never be able to fail an essential path. Feature flags let operators switch expensive optional features off during an incident.

## Proving it works

Resilience patterns that have never been exercised tend not to work when needed: a fallback with a typo, a timeout that's set but never applied, a breaker threshold nobody tuned. Test failure deliberately:

- **Fault injection** in staging and, carefully, in production: add latency or errors to one dependency and watch what happens (service meshes can do this per route).
- **Game days**: the team practises an incident — kill a database replica, block a region — while watching dashboards and runbooks.
- **Measure what users feel**: tie alerts to [SLOs](https://levelcode.org/concepts/sli-slo-sla) on error rate and latency, not to individual machine health.

## Practice

A news site's article page calls four services: `articles` (the text), `comments`, `related` (recommended articles) and `ads`. Last week `comments` slowed to 8-second responses during a database migration, and the whole site went down: article pages took 30 seconds, then failed. The page handler calls each service with the default HTTP client (no timeout), in sequence, retrying each call 3 times on error. All four share one HTTP connection pool of 100 connections.

Redesign the page's calls. Give concrete timeouts, retry rules, breaker and bulkhead settings, and the degraded behaviour for each dependency.

**Solution**

**Budget and parallelism.** Give the whole page a 1-second deadline with `context.WithTimeout`. Call `articles` first (essential), then call `comments`, `related` and `ads` **in parallel**, each with its own tighter deadline, so the slowest optional call doesn't add to the others.

**Per-dependency settings:**

| Service | Timeout | Retries | Breaker | Bulkhead | When it fails |
|---|---|---|---|---|---|
| articles | 400 ms | 1 retry with jittered backoff (GET is idempotent), within the deadline | Open at 50% failures over 20+ calls in 10 s, 15 s cool-down | Own pool, 60 connections | Serve the last cached copy from the CDN or cache; if there's none, a 503 page |
| comments | 200 ms | None | Same thresholds | 15 concurrent calls | Render the article without comments, with "Comments are unavailable" |
| related | 150 ms | None | Same | 15 | Show a static "most read" list |
| ads | 200 ms | None | Same | 15 | Render without ads (collapse the slots) |

**Why these choices.** Without timeouts, the 8-second `comments` calls held connections until the shared pool emptied, and `articles` calls queued behind them — that's what made the essential path fail. Separate pools (bulkheads) mean `comments` can exhaust only its own 15. The breaker makes `comments` fail in microseconds once it's clearly broken, so every page stops paying even the 200 ms. Removing retries from optional calls stops a slow dependency's load from tripling; one retry remains on `articles`, where a brief blip is worth hiding.

**Load shedding.** Add an in-flight limit at the front of the page service, returning 503 with `Retry-After` so the CDN can serve stale cached pages during overload.

**Verify** by injecting 8 seconds of latency into `comments` in staging and confirming article pages stay under a second with the comments section hidden.

## Key takeaways

- Cascading failures spread through shared resources and missing timeouts; contain failure where it starts.
- Every call needs a timeout based on the dependency's measured latency, and deadlines should shrink as a request travels.
- Retry only transient failures and only idempotent operations (or ones with idempotency keys), with exponential backoff, full jitter, a budget, and at one layer.
- Circuit breakers fail fast while a dependency is down and probe before trusting it again.
- Bulkheads give each dependency its own limited resources, so one can't exhaust everyone's.
- Shed load early with bounded queues, and decide in advance how each optional feature degrades.

## Quiz

1. A request passes through a gateway, an API and a database client, each retrying up to 3 attempts. The database is overloaded. How many database calls can one user request generate?

   A. 3
   B. 9
   C. 27
   D. 6

   **Answer: C.** Retries multiply across layers: 3 × 3 × 3 = 27. That amplification arrives exactly when the database is least able to handle it, which is why retries should live at one layer and be budgeted.

2. What does full jitter add to exponential backoff?

   A. It makes every client wait exactly the same time so retries are synchronised
   B. It picks a random wait up to the exponential cap, so clients that failed together don't retry together
   C. It removes the cap on the backoff
   D. It retries immediately on the first failure

   **Answer: B.** Without randomness, thousands of clients that failed at the same instant retry at the same instants, producing waves of load. Randomising the wait spreads them out.

3. A circuit breaker is open. What happens to calls?

   A. They queue until the dependency recovers
   B. They are retried with backoff
   C. They fail immediately without contacting the dependency, typically falling back to a default or cached response
   D. They are sent to the dependency with a longer timeout

   **Answer: C.** An open breaker stops calling a dependency that's clearly failing, so callers don't wait and the dependency gets room to recover. After a cool-down it goes half-open and lets trial calls through.

4. Under overload, why is a bounded queue with fast rejection better than an unbounded queue?

   A. Bounded queues use less CPU per request
   B. With an unbounded queue, wait times grow until every request times out, so the server does work nobody receives; rejection keeps latency bounded for the requests it accepts
   C. Unbounded queues lose messages
   D. Bounded queues guarantee every request eventually succeeds

   **Answer: B.** Once queueing delay exceeds client timeouts, all the work is wasted. Rejecting the excess early — load shedding and backpressure — keeps the accepted requests fast and signals callers to slow down.