Designing for failure

In a system of many services, something is always failing: a node is restarting, a disk is slow, a dependency is returning errors, a network link is dropping packets. That's normal, and users shouldn't notice. What turns normal failure into an outage is propagation — one slow component ties up the threads of its callers, whose callers then stall, until the whole system is waiting on one sick database. This lesson covers the patterns that contain failure, in the order requests meet them.

#How a cascading failure happens

A typical sequence:

1. The recommendations service's database gets slow: p99 goes from 20 ms to 4 seconds.
2. The product-page service calls recommendations with no timeout. Each request now holds a worker for 4 seconds instead of 50 ms.
3. By Little's law, in-flight requests grow 80-fold. The product-page service runs out of workers and connections.
4. Product pages time out at the load balancer. Clients retry. Load doubles.
5. The checkout service shares the product-page service's connection pool for price lookups. Checkout stops working.

The database problem was in a non-essential feature. The outage was in checkout. Every pattern below breaks one link in that chain.

#Timeouts and deadlines

Every network call needs a timeout. Without one, a hung dependency holds resources forever. See timeouts and deadlines.

Set timeouts from data, not instinct: a little above the dependency's p99.9 latency under normal load, and well below your own caller's patience. If a dependency normally answers in 30 ms at p99.9, a 200 ms timeout leaves room for noise without letting a stall last 30 seconds. Distinguish the connect timeout (short: a few hundred milliseconds) from the request timeout.

Better still, propagate a deadline. If the user's request has a 2-second budget and 1.5 seconds are gone, the next call should get at most 0.5 seconds — there's no point working on an answer nobody will wait for. Go's context does this naturally:

func (s *Server) handleProduct(w http.ResponseWriter, r *http.Request) {
	// The whole request gets 800 ms; every call below inherits what's left.
	ctx, cancel := context.WithTimeout(r.Context(), 800*time.Millisecond)
	defer cancel()

	product, err := s.catalog.Get(ctx, r.PathValue("id")) // essential
	if err != nil {
		http.Error(w, "unavailable", http.StatusServiceUnavailable)
		return
	}

	// Non-essential: a tighter budget of its own, and a fallback on failure.
	recCtx, recCancel := context.WithTimeout(ctx, 150*time.Millisecond)
	defer recCancel()
	recs, err := s.recs.For(recCtx, product.ID)
	if err != nil {
		recs = s.popularFallback() // degrade, don't fail
	}
	render(w, product, recs)
}

gRPC sends the deadline to the server in a header, so a downstream service can give up when its caller already has. For plain HTTP, pass the remaining budget in a header and apply it on the receiving side.

#Retries, without the stampede

Many failures are brief: a dropped connection, a node restarting, a leader election. A retry turns those into successes. Done carelessly, retries turn a struggling service into a dead one. See retries, backoff and jitter.

Retry only what's safe and might succeed. Retry timeouts, connection errors, 503 and 429. Don't retry 400 or 404 — the request is wrong and will stay wrong. Retry non-idempotent operations (creating an order, charging a card) only with an idempotency key the server uses to deduplicate.

Back off exponentially, with jitter. Wait longer after each attempt, and randomise the wait so thousands of clients that failed at the same moment don't all retry at the same moment. "Full jitter" — a random wait between zero and the exponential cap — spreads retries best:

// retry calls fn up to attempts times, sleeping a random duration in
// [0, min(maxDelay, base·2^n)) between tries, and stops when ctx is done.
func retry(ctx context.Context, attempts int, base, maxDelay time.Duration, fn func(context.Context) error) error {
	var err error
	for n := 0; n < attempts; n++ {
		if err = fn(ctx); err == nil || !retryable(err) {
			return err
		}
		if n == attempts-1 {
			break
		}
		backoff := min(maxDelay, base<<n)
		select {
		case <-time.After(rand.N(backoff)):
		case <-ctx.Done():
			return ctx.Err()
		}
	}
	return err
}

Cap retries at one layer. If the browser, the gateway, the API and the database client each make three attempts, one user action can produce 3 × 3 × 3 × 3 = 81 calls to the database while it's already overloaded. Retry at one layer — usually closest to the failure — and pass errors up from the others.

Use a retry budget. Allow retries to add at most, say, 10% to normal traffic. When most calls are failing, a retry is more likely to add load than to succeed, so the budget runs out and you stop retrying. Several gRPC and service-mesh implementations offer retry budgets built in.

#Circuit breakers

A timeout bounds how long one call waits. When a dependency is down, though, every call still waits the full timeout and then fails. A circuit breaker notices the pattern and stops calling for a while. See circuit breakers.

It has three states:

• Closed — normal. Calls pass through; failures are counted over a sliding window.
• Open — the failure rate crossed a threshold (for example 50% of at least 20 calls in 10 seconds). Calls fail immediately without touching the dependency, for a cool-down period. The dependency gets room to recover, and callers stop waiting.
• Half-open — after the cool-down, a few trial calls are let through. If they succeed, close; if not, open again.

Breakers pair with fallbacks: while open, return cached data, a default, or a reduced response. Libraries exist for every language (for example sony/gobreaker in Go), and service meshes such as Envoy implement the same idea as outlier detection.

#Bulkheads

Ships are divided into watertight compartments so one breach doesn't sink them. A bulkhead in software gives each dependency its own limited pool of resources, so a slow one can exhaust only its own share. See bulkheads.

In the cascading failure above, checkout died because it shared a pool with a feature that hung. With separate limits — at most 20 concurrent calls to recommendations, its own connection pool for prices — recommendations could only ever tie up 20 workers.

// A bulkhead: at most n concurrent calls; extra callers fail fast instead of queueing.
type Bulkhead struct{ slots chan struct{} }

func NewBulkhead(n int) *Bulkhead { return &Bulkhead{slots: make(chan struct{}, n)} }

var ErrBulkheadFull = errors.New("bulkhead full")

func (b *Bulkhead) Do(ctx context.Context, fn func(context.Context) error) error {
	select {
	case b.slots <- struct{}{}:
		defer func() { <-b.slots }()
		return fn(ctx)
	default:
		return ErrBulkheadFull
	}
}

The same idea applies at every scale: separate thread pools per dependency, separate database connection pools for user traffic and batch jobs, separate clusters for internal tools and customer traffic, and cell-based architectures where each "cell" serves a slice of customers so a bad deploy or poison request affects only that slice.

#Load shedding and backpressure

When demand exceeds capacity, something won't be served. You choose whether that's some requests, quickly rejected, or every request, slowly timing out. Load shedding chooses the former. See load shedding.

• Reject early and cheaply. Check before doing any work: if in-flight requests or queue wait time exceed a limit, return 503 with Retry-After. A rejection costs microseconds; a request that times out after 10 seconds of work wasted all of it.
• Shed by priority. Drop background and non-interactive traffic first, then anonymous traffic, and keep checkout and login last.
• Bound every queue. An unbounded queue under overload grows until requests wait longer than anyone will — every one then times out, and the server does work nobody receives. A bounded queue plus rejection is backpressure: it pushes the overload signal back to callers, who can slow down.

// shed rejects requests once more than limit are in flight, before any work is done.
func shed(limit int64, next http.Handler) http.Handler {
	var inflight atomic.Int64
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if inflight.Add(1) > limit {
			inflight.Add(-1)
			w.Header().Set("Retry-After", "1")
			http.Error(w, "overloaded", http.StatusServiceUnavailable)
			return
		}
		defer inflight.Add(-1)
		next.ServeHTTP(w, r)
	})
}

Rate limiting (previous lesson) caps each client. Load shedding caps the server, whatever the mix of clients. You want both.

#Graceful degradation

Decide in advance what the product does when each dependency is gone, and build it. See graceful degradation.

Classify features as essential (checkout, login) and optional (everything that decorates the page). Optional features get short timeouts, breakers and fallbacks; they must never be able to fail an essential path. Feature flags let operators switch expensive optional features off during an incident.

#Proving it works

Resilience patterns that have never been exercised tend not to work when needed: a fallback with a typo, a timeout that's set but never applied, a breaker threshold nobody tuned. Test failure deliberately:

• Fault injection in staging and, carefully, in production: add latency or errors to one dependency and watch what happens (service meshes can do this per route).
• Game days: the team practises an incident — kill a database replica, block a region — while watching dashboards and runbooks.
• Measure what users feel: tie alerts to SLOs on error rate and latency, not to individual machine health.

#Practice

A news site's article page calls four services: articles (the text), comments, related (recommended articles) and ads. Last week comments slowed to 8-second responses during a database migration, and the whole site went down: article pages took 30 seconds, then failed. The page handler calls each service with the default HTTP client (no timeout), in sequence, retrying each call 3 times on error. All four share one HTTP connection pool of 100 connections.

Redesign the page's calls. Give concrete timeouts, retry rules, breaker and bulkhead settings, and the degraded behaviour for each dependency.

#Key takeaways

• Cascading failures spread through shared resources and missing timeouts; contain failure where it starts.
• Every call needs a timeout based on the dependency's measured latency, and deadlines should shrink as a request travels.
• Retry only transient failures and only idempotent operations (or ones with idempotency keys), with exponential backoff, full jitter, a budget, and at one layer.
• Circuit breakers fail fast while a dependency is down and probe before trusting it again.
• Bulkheads give each dependency its own limited resources, so one can't exhaust everyone's.
• Shed load early with bounded queues, and decide in advance how each optional feature degrades.