Load shedding

When demand exceeds capacity, some requests won't be served well. The choice is between rejecting some quickly or serving all of them slowly — and "slowly" usually means everything times out, so nothing is served. Load shedding makes the first choice deliberately.

#Shed early and cheaply

A rejection should cost microseconds and happen before any expensive work: before parsing large bodies, querying the database or calling dependencies. A request that does 2 seconds of work and then times out at the client wasted all of it.

Common signals to shed on:

• Concurrency: more than N requests in flight (Little's law tells you N for your latency target).
• Queue wait time: a request has already waited longer than it could usefully wait (CoDel-style).
• Resource pressure: CPU or memory beyond a threshold.

Respond with 503 Service Unavailable (or 429 for per-client limits) and a Retry-After header.

#Shed by priority

Not all traffic is equal. Decide the order in advance:

1. Background jobs, prefetches, analytics beacons.
2. Anonymous and crawler traffic.
3. Non-critical features of logged-in sessions.
4. Last: checkout, login, payments.

Tag requests with a priority at the edge so every layer can make the same decision.

#Related mechanisms

• Rate limiting caps each client (rate-limiting algorithms); load shedding caps the server whatever the client mix. You want both.
• Backpressure pushes the overload signal upstream so producers slow down (backpressure).
• Autoscaling adds capacity, but takes minutes; shedding protects you in the meantime.
• Graceful degradation reduces the cost per request (turn off recommendations) so more requests fit (graceful degradation).

#Watch for retry storms

Shed requests come back as retries. Clients must back off with jitter, and Retry-After helps them do it (retries, backoff and jitter).