# API gateways > Put an API gateway in front of your services: route and version APIs, validate JWTs at the edge, rate-limit with token buckets, and avoid the classic anti-patterns. DevSecOps Engineer roadmap · Run the platform · 19 min read · https://levelcode.org/roadmaps/devsecops/api-gateways Once you have more than a couple of services, every one of them needs the same things at its front door: TLS, authentication, rate limits, request size limits, CORS, access logs. Implementing them in every service means implementing them inconsistently. An **API gateway** is a reverse proxy that does this work once, at the edge, for every API behind it. In this lesson you'll learn where a gateway sits relative to load balancers and service meshes, how it routes and versions APIs, how it validates JWTs and API keys (and why your backend must *still* authorise), how token-bucket rate limiting works with real numbers, and how to configure all of it with the Kubernetes Gateway API and Envoy Gateway. ## Gateway, load balancer or service mesh? All three are proxies, and they overlap. What differs is which traffic they handle and which questions they answer. | | Load balancer | API gateway | Service mesh | |---|---|---|---| | Traffic | Into a pool of identical backends | North-south: external clients into many APIs | East-west: service to service inside the cluster | | Main question | Which healthy backend gets this? | Who is calling, are they allowed this route, and how much? | Is this workload who it claims to be, and may it call that one? | | Typical features | Health checks, algorithms, TLS termination | Routing by path/host/header, JWT and API-key auth, rate limits, quotas, validation, CORS | mTLS between every pair of services, retries, traffic splitting, telemetry | | Identity it understands | None (IP addresses) | End users and client applications (tokens, keys) | Workloads (service accounts, certificates) | | Examples | AWS ALB/NLB, NGINX | Envoy Gateway, Kong, AWS API Gateway, Azure APIM, Apigee | Istio, Linkerd, Cilium | In practice they stack: a cloud L4 or L7 load balancer forwards to gateway Pods, the gateway authenticates and routes to services, and a mesh (the optional next lesson) secures the calls those services make to each other. Many gateways also load-balance across Pods, so the line between the first two columns is blurry; the gateway's distinguishing job is **API-level policy**. ## Routing and versioning The gateway maps a public URL space onto internal services. A request to `api.example.com/v1/orders/981` might go to the `orders-api` Service, `/v1/catalog` to a different team's service, and `/v2/orders` to a new implementation. Routing can match on: - **Host**: `api.example.com` versus `partners.example.com`. - **Path prefix or exact path**: the most common choice. - **Headers or query parameters**: `X-Api-Version: 2`, or sending internal testers to a canary. - **Method**: rarely needed, but possible. For **versioning**, a path prefix (`/v1/`, `/v2/`) is the easiest to route, cache and see in logs. Header-based versioning keeps URLs stable but is invisible in browser address bars and easy to forget in clients. Whichever you choose, the gateway lets you run old and new versions side by side, move traffic gradually with weighted backends, and keep a retired version returning `410 Gone` with a helpful message instead of a confusing 404. The Kubernetes **Gateway API** (`gateway.networking.k8s.io/v1`) splits this configuration by role. The platform team owns a `Gateway` (listeners, ports, certificates, which namespaces may attach routes); each application team owns `HTTPRoute` objects in its own namespace. That split is itself a security control: an app team can route its own paths but can't change the TLS settings or hijack another team's hostname. ## Authentication at the edge The gateway's most valuable job is rejecting unauthenticated requests before they consume backend resources. Three mechanisms cover nearly every API. ### JWT validation with JWKS Most APIs accept an OAuth 2.0 **access token** in the `Authorization: Bearer` header, and most identity providers issue it as a **JWT**: three base64url segments (header, payload, signature) separated by dots. Decoded, the first two look like this: ```json { "alg": "RS256", "typ": "JWT", "kid": "2026-09" } ``` ```json { "iss": "https://auth.example.com/", "sub": "user_7f3a", "aud": "orders-api", "scope": "orders.api profile", "iat": 1791712800, "exp": 1791713100 } ``` The identity provider signs tokens with a private key and publishes the matching public keys as a **JWKS** (JSON Web Key Set) at a URL such as `https://auth.example.com/.well-known/jwks.json`. The gateway fetches and caches that set, then validates every token without calling the identity provider: 1. **Algorithm**: accept only the algorithms you expect (here RS256 or ES256). Reject `none`, and never let a token pick HMAC when you hold an RSA public key: that "algorithm confusion" lets an attacker sign tokens with your public key as the secret. 2. **Key**: look up the `kid` in the cached JWKS. If it's unknown, refetch the set once (rate-limited) to pick up a key rotation, then reject. 3. **Signature**: verify it with that public key. 4. **Claims**: `iss` must equal your issuer exactly, `aud` must include your API, `exp` must be in the future and `nbf` (if present) in the past, allowing a minute or so of clock skew. 5. **Scope**: optionally require a coarse permission such as `orders.api`. Any failure returns `401 Unauthorized` with a `WWW-Authenticate: Bearer` header. A valid token without the required scope gets `403 Forbidden`. ### API keys An **API key** is a long random string that identifies a *client application* (a partner's integration, a mobile app build), not a user. Keys are useful for per-customer quotas, analytics and quickly cutting off one abusive integration. They're bearer secrets with no expiry and no user context, so treat them carefully: send them in a header, never in the URL (URLs end up in logs and browser history); store only a hash on the server side; scope each key to specific APIs; support rotation with two active keys per client. A key alone is not user authentication. ### OAuth 2.0 client credentials For machine-to-machine calls, the **client credentials** grant replaces long-lived API keys with short-lived tokens. The calling service authenticates to the identity provider and receives an access token, then calls the gateway with it like any other JWT: ```http POST /oauth2/token HTTP/1.1 Host: auth.example.com Authorization: Basic b3JkZXJzLXN5bmM6czNjcjN0 Content-Type: application/x-www-form-urlencoded grant_type=client_credentials&scope=orders.api ``` ```json { "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IjIwMjYtMDkifQ...", "token_type": "Bearer", "expires_in": 300 } ``` The client secret (here `orders-sync:s3cr3t`, base64-encoded in the Basic header) only ever goes to the identity provider, and a leaked access token is useless after five minutes. Stronger setups replace the secret with a signed assertion (`private_key_jwt`) or mutual TLS. ### Why the backend still authorises The gateway answers "is this a valid token for this API?". It can't answer "may `user_7f3a` read order 981?", because only the orders service knows who owns order 981. If the backend trusts anything that got past the gateway, any logged-in user can read everyone's orders by changing the ID: **broken object-level authorisation**, number one on the OWASP API Security Top 10. The gateway does authentication and coarse checks; the backend makes the fine-grained decision on every request. Two more reasons to keep checks in the backend: a new route added without its policy is unprotected, and internal callers may reach the service without going through the gateway. So make the gateway the only network path in (a NetworkPolicy that admits only the gateway's Pods, or mesh mTLS), and either validate the JWT again in the service (cheap with a cached JWKS) or accept identity headers *only* from the gateway, after the gateway has stripped any copy the client sent. ## Rate limiting and quotas Rate limits protect backends from overload and abuse, stop credential-stuffing on login endpoints, and turn "one client sent a million requests" from an outage into a few `429 Too Many Requests` responses. **Quotas** are the business version: 100,000 calls per month on the free plan, enforced per API key. ### The token bucket, worked through The most common algorithm is the **token bucket**. Each client has a bucket holding up to *capacity* tokens, refilled at a steady *rate*. Each request takes one token; if the bucket is empty, the request is rejected. Capacity sets the allowed burst and rate sets the long-run average. Take capacity 10 and refill 2 tokens per second (120 per minute): | Time | Event | Tokens before | Result | Tokens after | |---|---|---|---|---| | 0.00 s | Bucket starts full; client sends 12 requests at once | 10 | 10 allowed, 2 get 429 | 0 | | 1.00 s | 1 s elapsed, +2 tokens; client sends 3 | 2 | 2 allowed, 1 gets 429 | 0 | | 1.25 s | +0.5 tokens; client sends 1 | 0.5 | 429, `Retry-After: 1` (a token is due in 0.25 s) | 0.5 | | 7.00 s | +11.5 tokens, capped at capacity; client sends 1 | 10 | Allowed | 9 | Compare a **fixed window** counter ("100 per calendar minute"): a client can send 100 requests at 12:00:59 and 100 more at 12:01:00, 200 in two seconds. Token buckets and sliding windows smooth that boundary. Real gateways mix both: Envoy's local rate limiter is a token bucket, while its global rate limit service counts per fixed window in Redis. ### Per-key or per-IP? | Limit by | Good for | Weakness | |---|---|---| | API key or token subject | Authenticated APIs, fair use between customers, paid tiers | Doesn't protect unauthenticated endpoints | | Client IP | Login, sign-up, password reset, anonymous public endpoints | Many users share one IP behind corporate NAT or carrier-grade NAT; attackers rotate through thousands of IPs; IPv6 clients have whole /64 ranges | | Route or global | Protecting a fragile backend regardless of caller | One abuser can exhaust it for everyone | Use per-subject limits wherever you have identity and per-IP limits for anonymous and authentication endpoints, on top of a global ceiling per route. Per-IP limiting depends on knowing the real client IP, so it inherits everything from the load-balancing lesson about trusting `X-Forwarded-For` only from your own hops. Two operational details matter. Limits counted separately on each gateway replica multiply with the replica count (3 replicas × 100/min is really 300/min), so use a shared store for limits that must be exact. And decide in advance whether the gateway **fails open or closed** if that store is down: failing closed turns a Redis outage into an API outage. ## Validation, CORS, TLS and observability **Request validation and size limits.** Reject what can't be valid before it reaches code: bodies over a sane size (1 MiB is plenty for most JSON APIs), oversized headers and URLs, wrong content types, and, if you publish an OpenAPI description, requests that don't match its schema. Kong's `request-size-limiting` plugin, AWS API Gateway request validators and Azure APIM's `validate-content` policy all do this. Validation at the gateway reduces noise and attack surface; the backend still validates the fields it uses. **CORS.** Browsers send a preflight `OPTIONS` request before cross-origin calls with custom headers. Answer it at the gateway from an explicit allowlist of origins. Never reflect whatever `Origin` arrives while also sending `Access-Control-Allow-Credentials: true`: that lets any website make authenticated calls with your users' cookies. **TLS.** The gateway is usually where public TLS terminates, so it carries the TLS policy (1.2 minimum), the certificates, HSTS, and ideally TLS or mTLS onward to the backends. **Observability.** Because every request passes through it, the gateway is the best place for consistent access logs (route, status, latency, client ID, request ID), per-route metrics and the start of a distributed trace (`traceparent`). Log the token's subject, never the token itself, and redact `Authorization` and `Cookie` headers. ## Configuring a gateway: Gateway API with Envoy Gateway Envoy Gateway is a CNCF implementation of the Gateway API that adds its own policy resources for features the core API doesn't standardise (JWT, rate limits, CORS). Other implementations offer equivalent policies with different names. Step through the flow first. Look at where the two short-circuits happen: neither the expired token nor the over-limit request ever reaches the backend. ```sequence title: API gateway with JWT validation and rate limiting participants: Client, Gateway, Identity provider, Orders service Gateway -> Identity provider: GET /.well-known/jwks.json Identity provider --> Gateway: JWKS (kid 2026-09, 2026-06) > Fetched at startup and refreshed periodically, not per request. The identity provider is off the hot path. Client -> Gateway: GET /v1/orders/981 (Bearer token) Gateway -> Gateway: Verify kid 2026-09 signature, iss, aud, exp > Validation uses only the cached public key, so it costs microseconds. Gateway -> Gateway: Bucket for sub=user_7f3a: 7 tokens → 6 Gateway -> Orders service: GET /v1/orders/981 + x-jwt-sub > The gateway forwards the request with the verified subject in a header it controls. Orders service -> Orders service: Does user_7f3a own order 981? > Object-level authorisation happens here, because only this service knows who owns what. Orders service --> Gateway: 200 OK Gateway --> Client: 200 OK note Client, Gateway: Short-circuit 1: expired token Client -> Gateway: GET /v1/orders (exp in the past) Gateway --> Client: 401 Unauthorized, WWW-Authenticate: Bearer note Client, Gateway: Short-circuit 2: bucket empty Client -> Gateway: GET /v1/orders (valid token) Gateway --> Client: 429 Too Many Requests, Retry-After: 1 ``` The platform team's `Gateway`, plus a gateway-wide policy that fixes client IP detection and strips the identity header so clients can't forge it: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: public namespace: gateway-system spec: gatewayClassName: eg listeners: - name: https protocol: HTTPS port: 443 hostname: api.example.com tls: mode: Terminate certificateRefs: - kind: Secret name: api-example-com-tls allowedRoutes: namespaces: from: Selector selector: matchLabels: gateway-access: public --- apiVersion: gateway.envoyproxy.io/v1alpha1 kind: ClientTrafficPolicy metadata: name: public-clients namespace: gateway-system spec: targetRefs: - group: gateway.networking.k8s.io kind: Gateway name: public clientIPDetection: xForwardedFor: numTrustedHops: 1 headers: earlyRequestHeaders: remove: - x-jwt-sub ``` `numTrustedHops: 1` says one trusted proxy (the cloud load balancer) sits in front, so the gateway takes the client address from the right of `X-Forwarded-For` rather than the attacker-controlled left. The orders team's route and policies, in its own namespace (labelled `gateway-access: public`): ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: orders namespace: shop spec: parentRefs: - name: public namespace: gateway-system sectionName: https hostnames: - api.example.com rules: - matches: - path: type: PathPrefix value: /v1/orders backendRefs: - name: orders-api port: 8080 timeouts: request: 10s - matches: - path: type: PathPrefix value: /v2/orders backendRefs: - name: orders-api-v2 port: 8080 --- apiVersion: gateway.envoyproxy.io/v1alpha1 kind: SecurityPolicy metadata: name: orders-auth namespace: shop spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: orders jwt: providers: - name: corp-idp issuer: https://auth.example.com/ audiences: - orders-api remoteJWKS: uri: https://auth.example.com/.well-known/jwks.json claimToHeaders: - claim: sub header: x-jwt-sub authorization: defaultAction: Deny rules: - name: require-api-scope action: Allow principal: jwt: provider: corp-idp scopes: ["orders.api"] cors: allowOrigins: - https://app.example.com allowMethods: [GET, POST] allowHeaders: [authorization, content-type] maxAge: 600s --- apiVersion: gateway.envoyproxy.io/v1alpha1 kind: BackendTrafficPolicy metadata: name: orders-limits namespace: shop spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: orders rateLimit: global: rules: - clientSelectors: - headers: - name: x-jwt-sub type: Distinct limit: requests: 120 unit: Minute ``` Read it as one pipeline. The JWT provider validates issuer, audience, expiry and signature against the remote JWKS and copies the `sub` claim into `x-jwt-sub`. The authorization block denies any token lacking the `orders.api` scope (403). The rate limit keeps a separate counter for each distinct subject, 120 requests per minute. Global limits are shared across gateway replicas through Redis, which you enable once in the Envoy Gateway installation (`config.envoyGateway.rateLimit.backend.type=Redis` plus `config.envoyGateway.rateLimit.backend.redis.url=:6379` in its Helm values). ### Managed gateways If you'd rather not run the proxy yourself, every cloud sells one. **AWS API Gateway** offers HTTP APIs (with built-in JWT authorizers) and REST APIs (with usage plans, API keys, throttling and request validation). **Azure API Management** expresses the same features as XML policies such as `validate-jwt`, `rate-limit-by-key` and `quota-by-key`, plus a developer portal. **Google Apigee** is a full API management platform with analytics, monetisation and a portal; Google Cloud API Gateway is a lighter option. The concepts in this lesson map directly onto all of them. ## Anti-patterns - **Business logic in the gateway.** Pricing rules, order validation or response stitching scripted into gateway plugins are untestable, unversioned with the service, and owned by the wrong team. Keep the gateway to cross-cutting policy; put aggregation in a dedicated backend-for-frontend service. - **A single point of failure.** One gateway instance in one zone means one crash takes down every API. Run at least two replicas across zones with a PodDisruptionBudget, autoscale them, and set timeouts so one slow backend can't tie up every gateway connection. - **The gateway as the only security layer.** Backends reachable directly, or backends that trust any request carrying an `x-user-id` header, turn one bypass into total compromise. - **One giant shared config.** If every change goes through one team's monolithic file, teams work around it. The Gateway API's Gateway/HTTPRoute split exists to prevent this. - **Logging secrets.** Access logs full of bearer tokens and API keys are a credential leak waiting to happen. ## Practice Two parts: one in code, one in configuration. 1. Write a per-key token bucket limiter in Python with `allow(key, now) -> (allowed, retry_after_seconds)`. Using capacity 10 and 2 tokens per second, reproduce every row of the worked example table, and show that a second key has its own bucket. 2. Extend the Envoy Gateway configuration above for two new requirements: (a) a public, unauthenticated `GET /v1/catalog` route to the `catalog-api` Service (port 8080) in the same namespace, limited to 30 requests per minute **per client IP**; (b) send 10% of `/v1/orders` traffic to a canary Service `orders-api-canary`. **Solution** Part 1, `ratelimit.py`: ```python import math from dataclasses import dataclass @dataclass class Bucket: tokens: float updated: float class TokenBucketLimiter: def __init__(self, capacity: int, refill_per_sec: float) -> None: self.capacity = capacity self.rate = refill_per_sec self.buckets: dict[str, Bucket] = {} def allow(self, key: str, now: float) -> tuple[bool, float]: """Return (allowed, seconds until a token is available).""" b = self.buckets.get(key) if b is None: b = self.buckets[key] = Bucket(tokens=self.capacity, updated=now) b.tokens = min(self.capacity, b.tokens + (now - b.updated) * self.rate) b.updated = now if b.tokens >= 1: b.tokens -= 1 return True, 0.0 return False, (1 - b.tokens) / self.rate if __name__ == "__main__": limiter = TokenBucketLimiter(capacity=10, refill_per_sec=2) burst = [limiter.allow("alice", 0.0)[0] for _ in range(12)] print(f"t=0.00 alice x12: {burst.count(True)} allowed, {burst.count(False)} rejected") print("t=1.00 alice x3: ", [limiter.allow("alice", 1.0)[0] for _ in range(3)]) ok, wait = limiter.allow("alice", 1.25) print(f"t=1.25 alice: allowed={ok}, retry in {wait:.2f}s -> Retry-After: {math.ceil(wait)}") print("t=1.25 bob: ", limiter.allow("bob", 1.25)[0]) limiter.allow("alice", 7.0) print(f"t=7.00 alice: tokens left after 1 request = {limiter.buckets['alice'].tokens:.1f}") ``` ```text $ python3 ratelimit.py t=0.00 alice x12: 10 allowed, 2 rejected t=1.00 alice x3: [True, True, False] t=1.25 alice: allowed=False, retry in 0.25s -> Retry-After: 1 t=1.25 bob: True t=7.00 alice: tokens left after 1 request = 9.0 ``` The refill is computed lazily from elapsed time, so no background timer is needed, and `min(...)` caps the bucket at capacity, which is why 5.75 idle seconds don't bank more than 10 tokens. `Retry-After` takes whole seconds, so 0.25 s rounds up to 1. In a real gateway the buckets live in shared storage (or per replica, with the limit divided accordingly) and idle buckets are evicted. Part 2. The catalog needs its own HTTPRoute, because the JWT `SecurityPolicy` targets the whole `orders` route; a separate route gets its own, IP-based limit and no authentication: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: catalog namespace: shop spec: parentRefs: - name: public namespace: gateway-system sectionName: https hostnames: - api.example.com rules: - matches: - path: type: PathPrefix value: /v1/catalog method: GET backendRefs: - name: catalog-api port: 8080 --- apiVersion: gateway.envoyproxy.io/v1alpha1 kind: BackendTrafficPolicy metadata: name: catalog-limits namespace: shop spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: catalog rateLimit: global: rules: - clientSelectors: - sourceCIDR: type: Distinct value: 0.0.0.0/0 limit: requests: 30 unit: Minute ``` `sourceCIDR` with `type: Distinct` keeps one counter per client address inside the range. It uses the client IP as determined by the `ClientTrafficPolicy` (`numTrustedHops: 1`), which is why that policy matters: without it, every request would appear to come from the load balancer and all clients would share one bucket, or, with naive XFF parsing, each attacker request could claim a fresh IP. Add a second rule with `value: "::/0"` if the listener accepts IPv6. Because only `GET` matches, other methods on `/v1/catalog` get a 404 from the gateway. For the canary, give the existing `/v1/orders` rule two weighted backends; weights are relative, so 90 and 10 send about 10% of requests to the canary: ```yaml - matches: - path: type: PathPrefix value: /v1/orders backendRefs: - name: orders-api port: 8080 weight: 90 - name: orders-api-canary port: 8080 weight: 10 timeouts: request: 10s ``` The canary inherits the route's JWT, scope and rate-limit policies automatically, because they target the route rather than a backend. Watch its error rate and latency in the gateway's per-backend metrics before raising its weight. ## Key takeaways - A load balancer picks a healthy backend, a service mesh secures service-to-service calls, and an API gateway applies API-level policy (identity, routes, limits) to traffic entering from outside. - Validate JWTs at the edge against a cached JWKS: pin the algorithms, match `kid`, verify the signature, and check `iss`, `aud` and `exp`; use client credentials instead of long-lived keys for machines. - The gateway authenticates; the backend still authorises every object access, and it must be reachable only through the gateway. - A token bucket allows bursts up to its capacity and a long-run rate equal to its refill; limit per key or subject where you have identity, per IP for anonymous and login endpoints. - Enforce size limits, strict CORS allowlists and a TLS policy at the gateway, and log subjects, never tokens. - Keep business logic out of the gateway, run it redundantly across zones, and let the Gateway API split ownership between platform and app teams. ## Quiz 1. The gateway validates JWTs and requires the `orders.api` scope. A logged-in user changes `/v1/orders/981` to `/v1/orders/982` and sees another customer's order. Where is the bug? A. The gateway should have rejected the token B. The JWKS cache is stale C. The orders service doesn't check that the token's subject owns the requested order D. The rate limit is too high **Answer: C.** The token is valid and has the right scope, so the gateway correctly let it through. Only the backend knows who owns order 982; this is broken object-level authorisation and must be fixed in the service. 2. A token bucket has capacity 20 and refills 5 tokens per second. It is empty at t = 0. A client sends one request every 100 ms from t = 0 onwards. Roughly how many requests per second succeed once things settle? A. 10 B. 20 C. 2 D. 5 **Answer: D.** The client sends 10 per second but tokens arrive at 5 per second, and an empty bucket never builds up a burst. The long-run allowed rate equals the refill rate; capacity only sets the burst size. 3. Why should a gateway reject JWTs whose header says `"alg": "HS256"` when your identity provider signs with RS256? A. An attacker could sign a forged token with HMAC using your published public key as the secret, and a naive verifier would accept it B. HS256 tokens are always expired C. HS256 cannot carry a `kid` D. HS256 is slower to verify **Answer: A.** This is algorithm confusion. If the verifier lets the token choose the algorithm, it may run HMAC with the RSA public key, which the attacker also has. Pin the accepted algorithms in configuration. 4. Your gateway runs 4 replicas, each enforcing a local limit of 100 requests per minute per API key. A customer complains that a client is getting far more than 100 per minute. Why? A. Token buckets don't work with API keys B. Each replica counts separately, so a client spread across replicas can get up to about 400 per minute C. The gateway is failing closed D. CORS preflight requests aren't counted **Answer: B.** Local limits multiply with the replica count. Use a global limit backed by a shared store such as Redis, or divide the per-replica limit, when the number must be accurate.