Bulkheads

Ships are divided into watertight compartments — bulkheads — so a breach floods one compartment, not the whole hull. In software, a bulkhead gives each dependency or workload its own bounded share of resources, so when one misbehaves it exhausts only its share.

#The failure it prevents

A service has a pool of 200 worker threads and calls three dependencies. One dependency starts taking 10 seconds per call. Soon all 200 workers are waiting on it, and requests that don't even use that dependency can't get a worker. One slow dependency has taken down everything.

#Bulkheads at different scales

• Per-dependency concurrency limits: at most 30 in-flight calls to the recommendations service; extra calls fail fast or use a fallback.
• Separate connection pools: user traffic and batch jobs get different database pools, so a heavy report can't starve checkout.
• Separate thread pools or worker queues per kind of work: interactive requests versus background exports.
• Separate deployments or clusters: internal tools and customer traffic on different instances.
• Cells: split the whole system into independent cells, each serving a slice of customers, so a bad deploy or a poison request affects only one cell.

#A minimal concurrency bulkhead in Go

var sem = make(chan struct{}, 30) // at most 30 concurrent calls

func callRecommendations(ctx context.Context) ([]Item, error) {
	select {
	case sem <- struct{}{}:
		defer func() { <-sem }()
		return recs.Get(ctx)
	default:
		return popularItems(), nil // full: don't queue, degrade
	}
}

#Sizing

Use Little's law: a dependency handling 300 calls/s at 50 ms needs about 15 concurrent slots in normal operation; 30 leaves headroom while still capping the damage. Combine with timeouts and circuit breakers.