# Load balancing > Choose between L4 and L7 load balancers, pick an algorithm, configure health checks, TLS and draining, and keep the real client IP trustworthy. DevSecOps Engineer roadmap · Run the platform · 19 min read · https://levelcode.org/roadmaps/devsecops/load-balancing Every production service with more than one copy sits behind a load balancer. It decides which backend gets each request, stops sending traffic to broken backends, and is usually where TLS ends and the public internet stops: a reliability component and a security boundary at once. In this lesson you'll learn what L4 and L7 load balancers can see and do, how the common algorithms behave (with worked numbers), how health checks and draining keep deploys invisible, where TLS should terminate, how cloud and Kubernetes load balancers map onto these ideas, and how to keep the client IP honest. ## Why load balance A load balancer gives clients **one stable address** (a virtual IP or DNS name) in front of a pool of interchangeable **backends** (also called targets, upstreams or endpoints). That buys you two things: - **Scale.** When one instance can serve 800 requests per second and you need 3,000, you run four or five and spread the load. Adding capacity becomes "add a backend to the pool", which is exactly what autoscaling (the next lesson) automates. - **Availability.** If one backend crashes, is being deployed, or its zone loses power, the load balancer routes around it. Clients never learn that a machine went away. The second point only works if the load balancer *knows* a backend is unhealthy, and if backends can leave the pool without dropping requests mid-flight. ## L4 versus L7 Load balancers are classified by the highest network layer they understand. You met the layers in networking essentials: L4 is TCP/UDP (addresses and ports), L7 is the application protocol (HTTP, gRPC). | | Layer 4 (transport) | Layer 7 (application) | |---|---|---| | Sees | Source/destination IP and port, TCP flags | Everything in L4, plus method, path, host, headers, cookies, body | | Balances | Connections | Individual requests | | TLS | Passes encrypted bytes through (or terminates as a simple TLS listener) | Terminates TLS so it can read HTTP | | Can do | Fast, protocol-agnostic forwarding; any TCP/UDP protocol | Route `/api` and `/static` to different pools, retry failed requests, add headers, rewrite paths, enforce auth and WAF rules | | Examples | AWS NLB, Azure Load Balancer, IPVS, HAProxy in `mode tcp` | AWS ALB, NGINX, Envoy, HAProxy in `mode http` | An L4 balancer picks a backend when the TCP connection opens and sends every byte of that connection to the same place. An L7 balancer is a full **proxy**: it accepts the client's connection, reads each HTTP request, and sends it over a separate connection (often a pooled, kept-alive one) to the backend it chooses. That difference matters more than it looks. HTTP/2 and gRPC clients open *one* long-lived connection and multiplex thousands of requests over it. Behind an L4 balancer, that client is pinned to a single backend forever, and new backends added by autoscaling receive nothing from existing clients. An L7 balancer spreads the individual requests. Rule of thumb: use L7 for HTTP and gRPC, L4 for everything else (databases, MQTT, custom TCP), and L4 in front of your own L7 proxies when you need static IPs or extreme throughput. ## Balancing algorithms The algorithm decides which healthy backend gets the next connection or request. **Round robin** cycles through backends in order: A, B, C, A, B, C. It's the default almost everywhere and works well when backends are identical and requests cost about the same. **Weighted round robin** gives some backends a larger share, for a bigger instance or a canary that should get 10% of traffic. NGINX uses a "smooth" variant that interleaves picks instead of sending bursts. With weights A=5, B=1, C=1, each pick adds every backend's weight to its running score, chooses the highest score, then subtracts the total (7) from the winner: | Pick | Scores before pick (A, B, C) | Chosen | Scores after | |---|---|---|---| | 1 | 5, 1, 1 | A | −2, 1, 1 | | 2 | 3, 2, 2 | A | −4, 2, 2 | | 3 | 1, 3, 3 | B | 1, −4, 3 | | 4 | 6, −3, 4 | A | −1, −3, 4 | | 5 | 4, −2, 5 | C | 4, −2, −2 | | 6 | 9, −1, −1 | A | 2, −1, −1 | | 7 | 7, 0, 0 | A | 0, 0, 0 | The sequence is A A B A C A A: five of seven to A, and B and C are never hit back to back. **Least connections** sends the request to the backend with the fewest active connections. It shines when request cost varies: if B is stuck on three slow report exports, round robin keeps feeding it while least connections routes around it. Envoy's *least request* variant compares two randomly chosen backends, which stops many proxy instances stampeding onto the same "emptiest" one. **Hashing** picks the backend from a hash of something in the request (client IP, a header, the URL) so the same key always lands on the same backend. You want this for caches: requests for `/img/42.png` should hit the backend that already has it in memory. Run the same traffic through each algorithm below. Start with the uneven-work preset and switch between least connections and round robin while watching the p95 of normal requests, then try the degraded backend and the 5/1/1 weights. > **Interactive: Balancing algorithms, simulated.** Try it: https://levelcode.org/labs/load-balancer ### Consistent hashing, worked through The naive approach is `backend = hash(key) mod N`. It breaks badly when N changes. With three backends, remove the last one (C) and a key moves when `h mod 3 ≠ h mod 2`. Check the six possible residues of h modulo 6: only h = 0 and h = 1 keep their backend. Remove a different backend and the survivors are renumbered, so different residues survive, but still only two of the six. **Two thirds of all keys move**, and your cache hit rate collapses at exactly the moment you lost capacity. **Consistent hashing** places backends on a ring. Picture positions 0 to 99. Hash each backend name onto the ring: A lands at 20, B at 55, C at 85. To route a key, hash it and walk clockwise to the first backend: | Key | hash(key) | Walk clockwise to | Backend | |---|---|---|---| | `/img/7.png` | 12 | 20 | A | | `/img/42.png` | 37 | 55 | B | | `/img/9.png` | 61 | 85 | C | | `/img/3.png` | 93 | wraps to 20 | A | So A owns (85, 20], B owns (20, 55], C owns (55, 85]. Now remove B. Keys in (20, 55] walk on to C; **nothing else moves**. Only B's share, about a third of the keys, is remapped, and those had to move anyway. Adding a backend D at 40 steals only (20, 40] from B. Three points on a ring rarely split the space evenly (here A owns 35%, B 35%, C 30%, but with real hashes it could be 60/25/15). Implementations fix this with **virtual nodes**: each backend is hashed onto the ring 100 or more times, which evens out the shares. NGINX's `hash $request_uri consistent;` uses the ketama scheme, which does exactly that. Remove B from the ring below under modulo hashing and then under consistent hashing, and count the keys that move; then add D at 40 and switch on virtual nodes. > **Interactive: Modulo versus consistent hashing.** Try it: https://levelcode.org/labs/load-balancer ## Health checks A load balancer learns about broken backends in two ways. **Active health checks** probe each backend on a schedule: "GET `/readyz` every 5 seconds, 2-second timeout; 3 consecutive failures mark it unhealthy, 2 consecutive successes mark it healthy again." They detect a dead backend even when no traffic is flowing, and they let a recovered backend back in. Cloud load balancers and Envoy do this; open-source NGINX doesn't (active `health_check` is an NGINX Plus feature). **Passive health checks** (also called outlier detection) watch real traffic. If requests to a backend keep failing or timing out, it's taken out for a while. They react within a few requests, but only once users have already seen errors, and they can't tell when a backend recovers without sending it traffic again. Use both where you can. Step through a request to an L7 load balancer while one backend's health check fails. Watch for the moment B leaves the rotation: from then on, real requests only go to A, and B rejoins only after consecutive successes. ```sequence title: L7 load balancer removes a failing backend participants: Client, Load balancer, Backend A, Backend B Load balancer -> Backend A: GET /readyz Backend A --> Load balancer: 200 OK Load balancer -> Backend B: GET /readyz Backend B --> Load balancer: 503 (failure 3 of 3) > B lost its database connection pool. Its third consecutive failed check crosses the unhealthy threshold. Load balancer -> Load balancer: Mark B unhealthy, remove from pool > In-flight requests to B are allowed to finish; no new requests are sent to it. Client -> Load balancer: GET /orders/981 (TLS) > The balancer terminates TLS, reads the request, and chooses among healthy backends only. Load balancer -> Backend A: GET /orders/981 + X-Forwarded-For Backend A --> Load balancer: 200 OK Load balancer --> Client: 200 OK > The client never saw B's failure. Without the health check, roughly half of requests would have returned 503. Load balancer -> Backend B: GET /readyz (keeps probing) Backend B --> Load balancer: 200 OK (success 2 of 2) Load balancer -> Load balancer: Mark B healthy, add back to pool ``` In the simulation below, B crashes and recovers and then C starts failing half its requests. Compare the three presets: watch how long each takes to take B out, how many requests users pay for that, and how B gets back in. > **Interactive: Active and passive health checks.** Try it: https://levelcode.org/labs/load-balancer ### What the endpoint should check A health endpoint answers "should you send me traffic right now?" That's the **readiness** question. Kubernetes separates it from **liveness** ("am I so broken I should be restarted?"), as you saw with probes in Kubernetes fundamentals, and load balancer checks should map to readiness. Be careful how deep the check goes. If `/readyz` fails whenever the shared database is slow, a database blip fails *every* backend at once and the load balancer has nothing to route to (AWS ALB then "fails open" and sends traffic to all targets anyway). Check what is local to the instance (started, not shutting down, own thread pool not exhausted) and leave shared dependencies to monitoring. Keep the endpoint cheap and uninformative: no version strings or stack traces. ## TLS: terminate, pass through or re-encrypt Where TLS ends decides what the load balancer can do and what an attacker inside your network can see. | Mode | How it works | LB can read HTTP? | Traffic to backend | Use when | |---|---|---|---|---| | Termination | LB holds the certificate and decrypts | Yes | Plain HTTP | Backends are in a private, trusted network segment; simplest to operate | | Passthrough | LB forwards encrypted bytes; backend holds the certificate | No (only SNI from the ClientHello) | Original TLS | End-to-end encryption is mandatory, or the backend needs the client certificate itself | | Re-encryption (bridging) | LB decrypts, inspects, then opens a new TLS connection to the backend | Yes | New TLS session | You need L7 features *and* encryption on every hop | Passthrough forces L4 behaviour: no path routing, no header injection, no WAF. Termination is the common default, but "plain HTTP inside the VPC" means anyone with a foothold in that network can sniff tokens and cookies, so zero-trust designs re-encrypt, often with mutual TLS (the job of a service mesh, later in this stage). The TLS-terminating component sets the **TLS policy**: TLS 1.2 and 1.3 only, with forward-secret AEAD ciphers. Cloud load balancers express this as a named policy (on AWS, for example `ELBSecurityPolicy-TLS13-1-2-2021-06`, which allows only TLS 1.2 and 1.3; newer post-quantum variants exist, so check the current default); a policy that still allows TLS 1.0 should fail review. ## Sessions and connection draining **Sticky sessions** (session affinity) pin a client to one backend, usually with a cookie the load balancer sets (AWS ALB's `AWSALB`) or by hashing the client IP (`ip_hash` in NGINX). They exist because some apps keep session state in local memory. Avoid them: - Load becomes uneven: one heavy customer pins one backend, and new backends from a scale-out get only new clients. - A backend failure logs out everyone pinned to it, the opposite of high availability. - IP-based stickiness lumps whole offices behind one NAT address onto a single backend. Keep backends stateless instead: sessions in Redis or a database, or signed tokens. **Connection draining** (AWS calls it *deregistration delay*) is what makes deploys and scale-in invisible. When a backend is removed on purpose, the balancer stops sending it *new* requests but lets in-flight ones finish for a grace period. In Kubernetes there's a race to handle: when a Pod is deleted, its endpoint is marked terminating and the kubelet sends `SIGTERM` at roughly the same moment, while load balancers and kube-proxy on every node learn about the change a little later. If the app exits instantly, requests still in transit hit a closed port. The standard fix is a short pause before shutdown, then a graceful drain: ```yaml spec: terminationGracePeriodSeconds: 45 containers: - name: orders-api image: registry.example.com/orders-api:1.4.2 lifecycle: preStop: sleep: seconds: 10 ``` The built-in `sleep` action (Kubernetes 1.30+) needs no shell, which suits distroless images. After the sleep the app receives `SIGTERM`, stops accepting connections and finishes the ones it has; the grace period must cover the sleep plus the longest request. Set the cloud load balancer's deregistration delay to the same order of magnitude (AWS defaults to 300 seconds, which slows every rollout for no benefit). ## Load balancers in the cloud and in Kubernetes Every cloud offers both layers as managed services: | | AWS | Google Cloud | Azure | |---|---|---|---| | L7 (HTTP/gRPC) | Application Load Balancer (ALB) | Application Load Balancer (global or regional, external or internal) | Application Gateway (regional); Front Door (global edge) | | L4 (TCP/UDP) | Network Load Balancer (NLB): static IPs, very high throughput, client IP preserved | Network Load Balancer (passthrough or proxy) | Azure Load Balancer | | WAF attaches to | ALB (AWS WAF) | Application LB (Cloud Armor) | Application Gateway, Front Door | In Kubernetes, a `Service` of `type: LoadBalancer` asks the cloud provider integration to create an L4 load balancer that forwards to the Service's Pods: ```yaml apiVersion: v1 kind: Service metadata: name: edge-proxy namespace: edge spec: type: LoadBalancer externalTrafficPolicy: Local selector: app: edge-proxy ports: - name: https port: 443 targetPort: 8443 protocol: TCP ``` With the default `externalTrafficPolicy: Cluster`, traffic can arrive at any node and is forwarded (and source-NATed) to a Pod on another node, so the Pod sees a node IP instead of the client's. `Local` sends traffic only to nodes running a ready Pod and preserves the client IP, at the cost of possibly uneven spreading. An annotation makes the load balancer internal-only (`service.beta.kubernetes.io/aws-load-balancer-scheme: internal` on AWS), and `spec.loadBalancerSourceRanges` restricts which client CIDRs may connect. For HTTP, that L4 balancer usually fronts an **Ingress controller** or **Gateway API** implementation: an L7 proxy (Envoy, NGINX, HAProxy, Traefik) running as Pods, as you saw in Kubernetes fundamentals. Cloud controllers can instead program the cloud's own L7 balancer; the AWS Load Balancer Controller turns Ingress objects into ALBs that send traffic straight to Pod IPs. ## The security angle ### Preserving the client IP, honestly Behind an L7 proxy, the backend's TCP peer is the proxy. The original client address travels in a header, `X-Forwarded-For` (XFF), to which each proxy appends the address it received the connection from: ```http X-Forwarded-For: 203.0.113.7, 10.0.1.25 ``` The problem: **the client controls the first value**. If an attacker sends `X-Forwarded-For: 127.0.0.1` and your edge proxy appends to it, the backend sees `127.0.0.1, 198.51.100.9`. Code that reads the leftmost entry now believes the request came from localhost, which bypasses IP allowlists, poisons audit logs and defeats per-IP rate limits. The rule: **only trust entries added by proxies you operate**, counting from the right. With one proxy layer, the rightmost entry is the real client and everything to its left is untrusted input. Better still, the edge proxy overwrites the header instead of appending, as the config below does. L4 balancers can't add headers. Instead they either preserve the source IP at the packet level (NLB) or prepend a **PROXY protocol** header (v1 text, v2 binary) to the TCP stream, which carries the original address. The receiver must be configured to expect it (`listen 443 ssl proxy_protocol;` in NGINX), and must accept it **only** from the load balancer's addresses; otherwise anyone who can reach the port can claim any IP. ### Keeping backends private The load balancer should be the only way in. Put backends in private subnets with no public IPs and allow inbound traffic only from the load balancer's security group (in Kubernetes, a NetworkPolicy that admits only the ingress proxy's Pods). A directly reachable backend bypasses every control on the balancer: TLS policy, WAF, rate limits, auth. Block admin and debug paths such as `/actuator` at the L7 layer too. ### A real NGINX edge configuration This configuration puts the ideas together: a weighted least-connections pool with passive health checks and a backup, TLS termination with a modern policy, connection reuse to backends, an XFF header that the client can't forge, and bounded retries. ```nginx upstream orders_api { zone orders_api 64k; # share state (failure counts) across worker processes least_conn; server 10.0.1.11:8080 weight=2 max_fails=3 fail_timeout=10s; server 10.0.1.12:8080 max_fails=3 fail_timeout=10s; server 10.0.1.13:8080 max_fails=3 fail_timeout=10s; server 10.0.2.10:8080 backup; # only used when all the others are down keepalive 32; # idle connections kept open to backends } server { listen 443 ssl; http2 on; server_name api.example.com; ssl_certificate /etc/nginx/tls/api.example.com.fullchain.pem; ssl_certificate_key /etc/nginx/tls/api.example.com.key; ssl_protocols TLSv1.2 TLSv1.3; ssl_prefer_server_ciphers off; client_max_body_size 2m; location / { proxy_pass http://orders_api; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $remote_addr; # overwrite: this is the first hop proxy_set_header X-Forwarded-Proto https; proxy_connect_timeout 2s; proxy_read_timeout 30s; proxy_next_upstream error timeout http_502 http_503; proxy_next_upstream_tries 2; } location ~ ^/(actuator|metrics|debug)/ { return 404; } } ``` `max_fails=3 fail_timeout=10s` is NGINX's passive health check: three failed attempts within 10 seconds take a server out for 10 seconds. What counts as a failure is whatever `proxy_next_upstream` lists, so a 404 never marks a server down but a 502 does. Retries are capped at two tries, and NGINX never retries non-idempotent requests such as POST unless you add `non_idempotent`, which protects you from charging a card twice. `proxy_http_version 1.1` with an empty `Connection` header is what makes the `keepalive` pool actually work. > [!WARNING] > `$proxy_add_x_forwarded_for` appends to whatever the client sent. It's correct on an *inner* proxy that sits behind a trusted one, and wrong on the edge. On inner layers, use `set_real_ip_from ; real_ip_header X-Forwarded-For; real_ip_recursive on;` so NGINX skips trusted hops from the right and takes the first untrusted address as the client. ## Practice Build a three-backend pool on your machine with Docker Compose and break it on purpose. 1. Run three `traefik/whoami` containers (they echo their hostname and the request headers) behind NGINX, with weights 3, 1, 1 and passive health checks. 2. Send 10 requests and count which backend answered each. Predict the counts first. 3. Stop one backend and confirm requests keep succeeding. 4. Send a forged `X-Forwarded-For: 6.6.6.6` header and compare what the backend sees with `$proxy_add_x_forwarded_for` versus `$remote_addr`. **Solution** `compose.yaml`: ```yaml services: web1: image: traefik/whoami:v1.10.1 hostname: web1 web2: image: traefik/whoami:v1.10.1 hostname: web2 web3: image: traefik/whoami:v1.10.1 hostname: web3 lb: image: nginx:1.28-alpine ports: - "8080:80" volumes: - ./default.conf:/etc/nginx/conf.d/default.conf:ro depends_on: [web1, web2, web3] ``` `default.conf`: ```nginx upstream app { zone app 64k; server web1:80 weight=3 max_fails=2 fail_timeout=10s; server web2:80 max_fails=2 fail_timeout=10s; server web3:80 max_fails=2 fail_timeout=10s; keepalive 16; } server { listen 80; location / { proxy_pass http://app; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 1s; proxy_next_upstream error timeout http_502 http_503; } } ``` Steps 1 and 2. Smooth weighted round robin with weights 3, 1, 1 produces web1, web2, web1, web3, web1 per cycle of five, so 10 requests give 6/2/2: ```bash docker compose up -d for i in $(seq 10); do curl -s localhost:8080 | grep Hostname; done | sort | uniq -c ``` ```text 6 Hostname: web1 2 Hostname: web2 2 Hostname: web3 ``` Step 3. Stop a backend and repeat: ```bash docker compose stop web2 for i in $(seq 10); do curl -s -o /dev/null -w '%{http_code}\n' localhost:8080; done | sort | uniq -c ``` ```text 10 200 ``` Every request succeeds. Attempts to reach web2 fail, NGINX retries them on another server (`proxy_next_upstream error timeout`), and after two failures web2 is skipped for 10 seconds. A request or two may take up to the 1-second connect timeout: the cost of passive-only checks. `docker compose logs lb` shows the upstream errors. Step 4. With the appending variable, the forged value survives: ```bash docker compose start web2 curl -s -H 'X-Forwarded-For: 6.6.6.6' localhost:8080 | grep -i x-forwarded-for ``` ```text X-Forwarded-For: 6.6.6.6, 172.18.0.1 ``` A backend reading the leftmost entry would believe the client is `6.6.6.6`. Change the line to `proxy_set_header X-Forwarded-For $remote_addr;`, run `docker compose restart lb`, and repeat: ```text X-Forwarded-For: 172.18.0.1 ``` Now the header holds only the address NGINX actually saw (the Docker bridge gateway, because curl came through Docker's port mapping). That's the right setting for the first proxy a client reaches. ## Key takeaways - A load balancer gives clients one stable address over a pool of backends, for both scale and availability; it is also a security boundary. - L4 balances connections and sees only addresses and ports; L7 terminates TLS and balances individual requests, which you need for path routing, retries and long-lived HTTP/2 or gRPC connections. - Round robin suits uniform work, least connections suits uneven work, and consistent hashing keeps cache affinity while moving only about 1/N of keys when the pool changes. - Health checks should answer the readiness question and avoid failing every backend on a shared dependency; draining plus a `preStop` pause makes deploys invisible. - Avoid sticky sessions: keep backends stateless so any backend can serve any request. - Trust `X-Forwarded-For` and PROXY protocol only from proxies you run, enforce a TLS 1.2+ policy, and make the load balancer the only path to private backends. ## Quiz 1. A gRPC service runs behind an L4 load balancer. After autoscaling adds three new Pods, they receive almost no traffic. Why? A. L4 balancers only support round robin B. Each gRPC client multiplexes requests over one long-lived connection, and an L4 balancer picks a backend per connection, not per request C. The new Pods failed their liveness probes D. gRPC requires sticky sessions **Answer: B.** L4 balancing happens when the TCP connection opens. Existing HTTP/2 connections stay pinned to the old backends, so new backends only get new connections. An L7 balancer spreads individual requests. 2. A cache tier uses `hash(url) mod N` to pick one of 4 backends. One backend dies. What problem does consistent hashing solve here? A. It encrypts the cache keys B. It guarantees every backend gets exactly 25% of keys C. It makes the dead backend's keys fail instead of moving D. With modulo hashing most keys change backend when N changes; consistent hashing moves only the dead backend's share **Answer: D.** Changing N remaps most keys under modulo hashing, emptying every cache at once. On a hash ring, only keys owned by the removed backend move to its clockwise neighbour. 3. Your app reads the leftmost value of X-Forwarded-For to enforce an admin IP allowlist. You run one proxy layer that appends to the header. What can an attacker do? A. Send their own X-Forwarded-For header with an allowlisted IP and pass the check B. Nothing, because the proxy validates the header C. Only change the rightmost value D. Downgrade the TLS version **Answer: A.** Clients can send any X-Forwarded-For value; an appending proxy keeps it on the left. Only entries added by your own proxies, counted from the right, are trustworthy, or the edge proxy should overwrite the header. 4. Users see a burst of connection errors during every Kubernetes rollout, even though readiness probes are configured. What is the most likely fix? A. Enable sticky sessions B. Switch the load balancer from L7 to L4 C. Add a preStop sleep and graceful SIGTERM handling, with a grace period long enough to drain D. Make the readiness probe check the database **Answer: C.** Endpoint removal propagates to load balancers asynchronously while SIGTERM arrives immediately. A short preStop pause lets routing catch up, and graceful shutdown finishes in-flight requests.