Load balancing

Every production service with more than one copy sits behind a load balancer. It decides which backend gets each request, stops sending traffic to broken backends, and is usually where TLS ends and the public internet stops: a reliability component and a security boundary at once. In this lesson you'll learn what L4 and L7 load balancers can see and do, how the common algorithms behave (with worked numbers), how health checks and draining keep deploys invisible, where TLS should terminate, how cloud and Kubernetes load balancers map onto these ideas, and how to keep the client IP honest.

#Why load balance

A load balancer gives clients one stable address (a virtual IP or DNS name) in front of a pool of interchangeable backends (also called targets, upstreams or endpoints). That buys you two things:

• Scale. When one instance can serve 800 requests per second and you need 3,000, you run four or five and spread the load. Adding capacity becomes "add a backend to the pool", which is exactly what autoscaling (the next lesson) automates.
• Availability. If one backend crashes, is being deployed, or its zone loses power, the load balancer routes around it. Clients never learn that a machine went away.

The second point only works if the load balancer knows a backend is unhealthy, and if backends can leave the pool without dropping requests mid-flight.

#L4 versus L7

Load balancers are classified by the highest network layer they understand. You met the layers in networking essentials: L4 is TCP/UDP (addresses and ports), L7 is the application protocol (HTTP, gRPC).

An L4 balancer picks a backend when the TCP connection opens and sends every byte of that connection to the same place. An L7 balancer is a full proxy: it accepts the client's connection, reads each HTTP request, and sends it over a separate connection (often a pooled, kept-alive one) to the backend it chooses.

That difference matters more than it looks. HTTP/2 and gRPC clients open one long-lived connection and multiplex thousands of requests over it. Behind an L4 balancer, that client is pinned to a single backend forever, and new backends added by autoscaling receive nothing from existing clients. An L7 balancer spreads the individual requests. Rule of thumb: use L7 for HTTP and gRPC, L4 for everything else (databases, MQTT, custom TCP), and L4 in front of your own L7 proxies when you need static IPs or extreme throughput.

#Balancing algorithms

The algorithm decides which healthy backend gets the next connection or request.

Round robin cycles through backends in order: A, B, C, A, B, C. It's the default almost everywhere and works well when backends are identical and requests cost about the same.

Weighted round robin gives some backends a larger share, for a bigger instance or a canary that should get 10% of traffic. NGINX uses a "smooth" variant that interleaves picks instead of sending bursts. With weights A=5, B=1, C=1, each pick adds every backend's weight to its running score, chooses the highest score, then subtracts the total (7) from the winner:

The sequence is A A B A C A A: five of seven to A, and B and C are never hit back to back.

Least connections sends the request to the backend with the fewest active connections. It shines when request cost varies: if B is stuck on three slow report exports, round robin keeps feeding it while least connections routes around it. Envoy's least request variant compares two randomly chosen backends, which stops many proxy instances stampeding onto the same "emptiest" one.

Hashing picks the backend from a hash of something in the request (client IP, a header, the URL) so the same key always lands on the same backend. You want this for caches: requests for /img/42.png should hit the backend that already has it in memory.

Run the same traffic through each algorithm below. Start with the uneven-work preset and switch between least connections and round robin while watching the p95 of normal requests, then try the degraded backend and the 5/1/1 weights.

#Consistent hashing, worked through

The naive approach is backend = hash(key) mod N. It breaks badly when N changes. With three backends, remove the last one (C) and a key moves when h mod 3 ≠ h mod 2. Check the six possible residues of h modulo 6: only h = 0 and h = 1 keep their backend. Remove a different backend and the survivors are renumbered, so different residues survive, but still only two of the six. Two thirds of all keys move, and your cache hit rate collapses at exactly the moment you lost capacity.

Consistent hashing places backends on a ring. Picture positions 0 to 99. Hash each backend name onto the ring: A lands at 20, B at 55, C at 85. To route a key, hash it and walk clockwise to the first backend:

So A owns (85, 20], B owns (20, 55], C owns (55, 85]. Now remove B. Keys in (20, 55] walk on to C; nothing else moves. Only B's share, about a third of the keys, is remapped, and those had to move anyway. Adding a backend D at 40 steals only (20, 40] from B.

Three points on a ring rarely split the space evenly (here A owns 35%, B 35%, C 30%, but with real hashes it could be 60/25/15). Implementations fix this with virtual nodes: each backend is hashed onto the ring 100 or more times, which evens out the shares. NGINX's hash $request_uri consistent; uses the ketama scheme, which does exactly that.

Remove B from the ring below under modulo hashing and then under consistent hashing, and count the keys that move; then add D at 40 and switch on virtual nodes.

#Health checks

A load balancer learns about broken backends in two ways.

Active health checks probe each backend on a schedule: "GET /readyz every 5 seconds, 2-second timeout; 3 consecutive failures mark it unhealthy, 2 consecutive successes mark it healthy again." They detect a dead backend even when no traffic is flowing, and they let a recovered backend back in. Cloud load balancers and Envoy do this; open-source NGINX doesn't (active health_check is an NGINX Plus feature).

Passive health checks (also called outlier detection) watch real traffic. If requests to a backend keep failing or timing out, it's taken out for a while. They react within a few requests, but only once users have already seen errors, and they can't tell when a backend recovers without sending it traffic again. Use both where you can.

Step through a request to an L7 load balancer while one backend's health check fails. Watch for the moment B leaves the rotation: from then on, real requests only go to A, and B rejoins only after consecutive successes.

In the simulation below, B crashes and recovers and then C starts failing half its requests. Compare the three presets: watch how long each takes to take B out, how many requests users pay for that, and how B gets back in.

#What the endpoint should check

A health endpoint answers "should you send me traffic right now?" That's the readiness question. Kubernetes separates it from liveness ("am I so broken I should be restarted?"), as you saw with probes in Kubernetes fundamentals, and load balancer checks should map to readiness.

Be careful how deep the check goes. If /readyz fails whenever the shared database is slow, a database blip fails every backend at once and the load balancer has nothing to route to (AWS ALB then "fails open" and sends traffic to all targets anyway). Check what is local to the instance (started, not shutting down, own thread pool not exhausted) and leave shared dependencies to monitoring. Keep the endpoint cheap and uninformative: no version strings or stack traces.

#TLS: terminate, pass through or re-encrypt

Where TLS ends decides what the load balancer can do and what an attacker inside your network can see.

Passthrough forces L4 behaviour: no path routing, no header injection, no WAF. Termination is the common default, but "plain HTTP inside the VPC" means anyone with a foothold in that network can sniff tokens and cookies, so zero-trust designs re-encrypt, often with mutual TLS (the job of a service mesh, later in this stage).

The TLS-terminating component sets the TLS policy: TLS 1.2 and 1.3 only, with forward-secret AEAD ciphers. Cloud load balancers express this as a named policy (on AWS, for example ELBSecurityPolicy-TLS13-1-2-2021-06, which allows only TLS 1.2 and 1.3; newer post-quantum variants exist, so check the current default); a policy that still allows TLS 1.0 should fail review.

#Sessions and connection draining

Sticky sessions (session affinity) pin a client to one backend, usually with a cookie the load balancer sets (AWS ALB's AWSALB) or by hashing the client IP (ip_hash in NGINX). They exist because some apps keep session state in local memory. Avoid them:

• Load becomes uneven: one heavy customer pins one backend, and new backends from a scale-out get only new clients.
• A backend failure logs out everyone pinned to it, the opposite of high availability.
• IP-based stickiness lumps whole offices behind one NAT address onto a single backend.

Keep backends stateless instead: sessions in Redis or a database, or signed tokens.

Connection draining (AWS calls it deregistration delay) is what makes deploys and scale-in invisible. When a backend is removed on purpose, the balancer stops sending it new requests but lets in-flight ones finish for a grace period. In Kubernetes there's a race to handle: when a Pod is deleted, its endpoint is marked terminating and the kubelet sends SIGTERM at roughly the same moment, while load balancers and kube-proxy on every node learn about the change a little later. If the app exits instantly, requests still in transit hit a closed port. The standard fix is a short pause before shutdown, then a graceful drain:

spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: orders-api
      image: registry.example.com/orders-api:1.4.2
      lifecycle:
        preStop:
          sleep:
            seconds: 10

The built-in sleep action (Kubernetes 1.30+) needs no shell, which suits distroless images. After the sleep the app receives SIGTERM, stops accepting connections and finishes the ones it has; the grace period must cover the sleep plus the longest request. Set the cloud load balancer's deregistration delay to the same order of magnitude (AWS defaults to 300 seconds, which slows every rollout for no benefit).

#Load balancers in the cloud and in Kubernetes

Every cloud offers both layers as managed services:

In Kubernetes, a Service of type: LoadBalancer asks the cloud provider integration to create an L4 load balancer that forwards to the Service's Pods:

apiVersion: v1
kind: Service
metadata:
  name: edge-proxy
  namespace: edge
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    app: edge-proxy
  ports:
    - name: https
      port: 443
      targetPort: 8443
      protocol: TCP

With the default externalTrafficPolicy: Cluster, traffic can arrive at any node and is forwarded (and source-NATed) to a Pod on another node, so the Pod sees a node IP instead of the client's. Local sends traffic only to nodes running a ready Pod and preserves the client IP, at the cost of possibly uneven spreading. An annotation makes the load balancer internal-only (service.beta.kubernetes.io/aws-load-balancer-scheme: internal on AWS), and spec.loadBalancerSourceRanges restricts which client CIDRs may connect.

For HTTP, that L4 balancer usually fronts an Ingress controller or Gateway API implementation: an L7 proxy (Envoy, NGINX, HAProxy, Traefik) running as Pods, as you saw in Kubernetes fundamentals. Cloud controllers can instead program the cloud's own L7 balancer; the AWS Load Balancer Controller turns Ingress objects into ALBs that send traffic straight to Pod IPs.

#The security angle

#Preserving the client IP, honestly

Behind an L7 proxy, the backend's TCP peer is the proxy. The original client address travels in a header, X-Forwarded-For (XFF), to which each proxy appends the address it received the connection from:

X-Forwarded-For: 203.0.113.7, 10.0.1.25

The problem: the client controls the first value. If an attacker sends X-Forwarded-For: 127.0.0.1 and your edge proxy appends to it, the backend sees 127.0.0.1, 198.51.100.9. Code that reads the leftmost entry now believes the request came from localhost, which bypasses IP allowlists, poisons audit logs and defeats per-IP rate limits. The rule: only trust entries added by proxies you operate, counting from the right. With one proxy layer, the rightmost entry is the real client and everything to its left is untrusted input. Better still, the edge proxy overwrites the header instead of appending, as the config below does.

L4 balancers can't add headers. Instead they either preserve the source IP at the packet level (NLB) or prepend a PROXY protocol header (v1 text, v2 binary) to the TCP stream, which carries the original address. The receiver must be configured to expect it (listen 443 ssl proxy_protocol; in NGINX), and must accept it only from the load balancer's addresses; otherwise anyone who can reach the port can claim any IP.

#Keeping backends private

The load balancer should be the only way in. Put backends in private subnets with no public IPs and allow inbound traffic only from the load balancer's security group (in Kubernetes, a NetworkPolicy that admits only the ingress proxy's Pods). A directly reachable backend bypasses every control on the balancer: TLS policy, WAF, rate limits, auth. Block admin and debug paths such as /actuator at the L7 layer too.

#A real NGINX edge configuration

This configuration puts the ideas together: a weighted least-connections pool with passive health checks and a backup, TLS termination with a modern policy, connection reuse to backends, an XFF header that the client can't forge, and bounded retries.

upstream orders_api {
    zone orders_api 64k;          # share state (failure counts) across worker processes
    least_conn;

    server 10.0.1.11:8080 weight=2 max_fails=3 fail_timeout=10s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.13:8080 max_fails=3 fail_timeout=10s;
    server 10.0.2.10:8080 backup;  # only used when all the others are down

    keepalive 32;                  # idle connections kept open to backends
}

server {
    listen 443 ssl;
    http2 on;
    server_name api.example.com;

    ssl_certificate     /etc/nginx/tls/api.example.com.fullchain.pem;
    ssl_certificate_key /etc/nginx/tls/api.example.com.key;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;

    client_max_body_size 2m;

    location / {
        proxy_pass http://orders_api;
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        proxy_set_header Host              $host;
        proxy_set_header X-Forwarded-For   $remote_addr;   # overwrite: this is the first hop
        proxy_set_header X-Forwarded-Proto https;

        proxy_connect_timeout 2s;
        proxy_read_timeout    30s;
        proxy_next_upstream error timeout http_502 http_503;
        proxy_next_upstream_tries 2;
    }

    location ~ ^/(actuator|metrics|debug)/ {
        return 404;
    }
}

max_fails=3 fail_timeout=10s is NGINX's passive health check: three failed attempts within 10 seconds take a server out for 10 seconds. What counts as a failure is whatever proxy_next_upstream lists, so a 404 never marks a server down but a 502 does. Retries are capped at two tries, and NGINX never retries non-idempotent requests such as POST unless you add non_idempotent, which protects you from charging a card twice. proxy_http_version 1.1 with an empty Connection header is what makes the keepalive pool actually work.

#Practice

Build a three-backend pool on your machine with Docker Compose and break it on purpose.

1. Run three traefik/whoami containers (they echo their hostname and the request headers) behind NGINX, with weights 3, 1, 1 and passive health checks.
2. Send 10 requests and count which backend answered each. Predict the counts first.
3. Stop one backend and confirm requests keep succeeding.
4. Send a forged X-Forwarded-For: 6.6.6.6 header and compare what the backend sees with $proxy_add_x_forwarded_for versus $remote_addr.

services:
  web1:
    image: traefik/whoami:v1.10.1
    hostname: web1
  web2:
    image: traefik/whoami:v1.10.1
    hostname: web2
  web3:
    image: traefik/whoami:v1.10.1
    hostname: web3
  lb:
    image: nginx:1.28-alpine
    ports:
      - "8080:80"
    volumes:
      - ./default.conf:/etc/nginx/conf.d/default.conf:ro
    depends_on: [web1, web2, web3]

upstream app {
    zone app 64k;
    server web1:80 weight=3 max_fails=2 fail_timeout=10s;
    server web2:80 max_fails=2 fail_timeout=10s;
    server web3:80 max_fails=2 fail_timeout=10s;
    keepalive 16;
}

server {
    listen 80;
    location / {
        proxy_pass http://app;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_connect_timeout 1s;
        proxy_next_upstream error timeout http_502 http_503;
    }
}

docker compose up -d
for i in $(seq 10); do curl -s localhost:8080 | grep Hostname; done | sort | uniq -c

      6 Hostname: web1
      2 Hostname: web2
      2 Hostname: web3

docker compose stop web2
for i in $(seq 10); do curl -s -o /dev/null -w '%{http_code}\n' localhost:8080; done | sort | uniq -c

     10 200

docker compose start web2
curl -s -H 'X-Forwarded-For: 6.6.6.6' localhost:8080 | grep -i x-forwarded-for

X-Forwarded-For: 6.6.6.6, 172.18.0.1

X-Forwarded-For: 172.18.0.1

#Key takeaways

• A load balancer gives clients one stable address over a pool of backends, for both scale and availability; it is also a security boundary.
• L4 balances connections and sees only addresses and ports; L7 terminates TLS and balances individual requests, which you need for path routing, retries and long-lived HTTP/2 or gRPC connections.
• Round robin suits uniform work, least connections suits uneven work, and consistent hashing keeps cache affinity while moving only about 1/N of keys when the pool changes.
• Health checks should answer the readiness question and avoid failing every backend on a shared dependency; draining plus a preStop pause makes deploys invisible.
• Avoid sticky sessions: keep backends stateless so any backend can serve any request.
• Trust X-Forwarded-For and PROXY protocol only from proxies you run, enforce a TLS 1.2+ policy, and make the load balancer the only path to private backends.