API gateways

Once you have more than a couple of services, every one of them needs the same things at its front door: TLS, authentication, rate limits, request size limits, CORS, access logs. Implementing them in every service means implementing them inconsistently. An API gateway is a reverse proxy that does this work once, at the edge, for every API behind it. In this lesson you'll learn where a gateway sits relative to load balancers and service meshes, how it routes and versions APIs, how it validates JWTs and API keys (and why your backend must still authorise), how token-bucket rate limiting works with real numbers, and how to configure all of it with the Kubernetes Gateway API and Envoy Gateway.

#Gateway, load balancer or service mesh?

All three are proxies, and they overlap. What differs is which traffic they handle and which questions they answer.

In practice they stack: a cloud L4 or L7 load balancer forwards to gateway Pods, the gateway authenticates and routes to services, and a mesh (the optional next lesson) secures the calls those services make to each other. Many gateways also load-balance across Pods, so the line between the first two columns is blurry; the gateway's distinguishing job is API-level policy.

#Routing and versioning

The gateway maps a public URL space onto internal services. A request to api.example.com/v1/orders/981 might go to the orders-api Service, /v1/catalog to a different team's service, and /v2/orders to a new implementation. Routing can match on:

• Host: api.example.com versus partners.example.com.
• Path prefix or exact path: the most common choice.
• Headers or query parameters: X-Api-Version: 2, or sending internal testers to a canary.
• Method: rarely needed, but possible.

For versioning, a path prefix (/v1/, /v2/) is the easiest to route, cache and see in logs. Header-based versioning keeps URLs stable but is invisible in browser address bars and easy to forget in clients. Whichever you choose, the gateway lets you run old and new versions side by side, move traffic gradually with weighted backends, and keep a retired version returning 410 Gone with a helpful message instead of a confusing 404.

The Kubernetes Gateway API (gateway.networking.k8s.io/v1) splits this configuration by role. The platform team owns a Gateway (listeners, ports, certificates, which namespaces may attach routes); each application team owns HTTPRoute objects in its own namespace. That split is itself a security control: an app team can route its own paths but can't change the TLS settings or hijack another team's hostname.

#Authentication at the edge

The gateway's most valuable job is rejecting unauthenticated requests before they consume backend resources. Three mechanisms cover nearly every API.

#JWT validation with JWKS

Most APIs accept an OAuth 2.0 access token in the Authorization: Bearer header, and most identity providers issue it as a JWT: three base64url segments (header, payload, signature) separated by dots. Decoded, the first two look like this:

{ "alg": "RS256", "typ": "JWT", "kid": "2026-09" }

{
  "iss": "https://auth.example.com/",
  "sub": "user_7f3a",
  "aud": "orders-api",
  "scope": "orders.api profile",
  "iat": 1791712800,
  "exp": 1791713100
}

The identity provider signs tokens with a private key and publishes the matching public keys as a JWKS (JSON Web Key Set) at a URL such as https://auth.example.com/.well-known/jwks.json. The gateway fetches and caches that set, then validates every token without calling the identity provider:

1. Algorithm: accept only the algorithms you expect (here RS256 or ES256). Reject none, and never let a token pick HMAC when you hold an RSA public key: that "algorithm confusion" lets an attacker sign tokens with your public key as the secret.
2. Key: look up the kid in the cached JWKS. If it's unknown, refetch the set once (rate-limited) to pick up a key rotation, then reject.
3. Signature: verify it with that public key.
4. Claims: iss must equal your issuer exactly, aud must include your API, exp must be in the future and nbf (if present) in the past, allowing a minute or so of clock skew.
5. Scope: optionally require a coarse permission such as orders.api.

Any failure returns 401 Unauthorized with a WWW-Authenticate: Bearer header. A valid token without the required scope gets 403 Forbidden.

#API keys

An API key is a long random string that identifies a client application (a partner's integration, a mobile app build), not a user. Keys are useful for per-customer quotas, analytics and quickly cutting off one abusive integration. They're bearer secrets with no expiry and no user context, so treat them carefully: send them in a header, never in the URL (URLs end up in logs and browser history); store only a hash on the server side; scope each key to specific APIs; support rotation with two active keys per client. A key alone is not user authentication.

#OAuth 2.0 client credentials

For machine-to-machine calls, the client credentials grant replaces long-lived API keys with short-lived tokens. The calling service authenticates to the identity provider and receives an access token, then calls the gateway with it like any other JWT:

POST /oauth2/token HTTP/1.1
Host: auth.example.com
Authorization: Basic b3JkZXJzLXN5bmM6czNjcjN0
Content-Type: application/x-www-form-urlencoded

grant_type=client_credentials&scope=orders.api

{ "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IjIwMjYtMDkifQ...", "token_type": "Bearer", "expires_in": 300 }

The client secret (here orders-sync:s3cr3t, base64-encoded in the Basic header) only ever goes to the identity provider, and a leaked access token is useless after five minutes. Stronger setups replace the secret with a signed assertion (private_key_jwt) or mutual TLS.

#Why the backend still authorises

The gateway answers "is this a valid token for this API?". It can't answer "may user_7f3a read order 981?", because only the orders service knows who owns order 981. If the backend trusts anything that got past the gateway, any logged-in user can read everyone's orders by changing the ID: broken object-level authorisation, number one on the OWASP API Security Top 10. The gateway does authentication and coarse checks; the backend makes the fine-grained decision on every request.

Two more reasons to keep checks in the backend: a new route added without its policy is unprotected, and internal callers may reach the service without going through the gateway. So make the gateway the only network path in (a NetworkPolicy that admits only the gateway's Pods, or mesh mTLS), and either validate the JWT again in the service (cheap with a cached JWKS) or accept identity headers only from the gateway, after the gateway has stripped any copy the client sent.

#Rate limiting and quotas

Rate limits protect backends from overload and abuse, stop credential-stuffing on login endpoints, and turn "one client sent a million requests" from an outage into a few 429 Too Many Requests responses. Quotas are the business version: 100,000 calls per month on the free plan, enforced per API key.

#The token bucket, worked through

The most common algorithm is the token bucket. Each client has a bucket holding up to capacity tokens, refilled at a steady rate. Each request takes one token; if the bucket is empty, the request is rejected. Capacity sets the allowed burst and rate sets the long-run average.

Take capacity 10 and refill 2 tokens per second (120 per minute):

Compare a fixed window counter ("100 per calendar minute"): a client can send 100 requests at 12:00:59 and 100 more at 12:01:00, 200 in two seconds. Token buckets and sliding windows smooth that boundary. Real gateways mix both: Envoy's local rate limiter is a token bucket, while its global rate limit service counts per fixed window in Redis.

#Per-key or per-IP?

Use per-subject limits wherever you have identity and per-IP limits for anonymous and authentication endpoints, on top of a global ceiling per route. Per-IP limiting depends on knowing the real client IP, so it inherits everything from the load-balancing lesson about trusting X-Forwarded-For only from your own hops.

Two operational details matter. Limits counted separately on each gateway replica multiply with the replica count (3 replicas × 100/min is really 300/min), so use a shared store for limits that must be exact. And decide in advance whether the gateway fails open or closed if that store is down: failing closed turns a Redis outage into an API outage.

#Validation, CORS, TLS and observability

Request validation and size limits. Reject what can't be valid before it reaches code: bodies over a sane size (1 MiB is plenty for most JSON APIs), oversized headers and URLs, wrong content types, and, if you publish an OpenAPI description, requests that don't match its schema. Kong's request-size-limiting plugin, AWS API Gateway request validators and Azure APIM's validate-content policy all do this. Validation at the gateway reduces noise and attack surface; the backend still validates the fields it uses.

CORS. Browsers send a preflight OPTIONS request before cross-origin calls with custom headers. Answer it at the gateway from an explicit allowlist of origins. Never reflect whatever Origin arrives while also sending Access-Control-Allow-Credentials: true: that lets any website make authenticated calls with your users' cookies.

TLS. The gateway is usually where public TLS terminates, so it carries the TLS policy (1.2 minimum), the certificates, HSTS, and ideally TLS or mTLS onward to the backends.

Observability. Because every request passes through it, the gateway is the best place for consistent access logs (route, status, latency, client ID, request ID), per-route metrics and the start of a distributed trace (traceparent). Log the token's subject, never the token itself, and redact Authorization and Cookie headers.

#Configuring a gateway: Gateway API with Envoy Gateway

Envoy Gateway is a CNCF implementation of the Gateway API that adds its own policy resources for features the core API doesn't standardise (JWT, rate limits, CORS). Other implementations offer equivalent policies with different names. Step through the flow first. Look at where the two short-circuits happen: neither the expired token nor the over-limit request ever reaches the backend.

The platform team's Gateway, plus a gateway-wide policy that fixes client IP detection and strips the identity header so clients can't forge it:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: public
  namespace: gateway-system
spec:
  gatewayClassName: eg
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: api.example.com
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: api-example-com-tls
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-access: public
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: public-clients
  namespace: gateway-system
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: public
  clientIPDetection:
    xForwardedFor:
      numTrustedHops: 1
  headers:
    earlyRequestHeaders:
      remove:
        - x-jwt-sub

numTrustedHops: 1 says one trusted proxy (the cloud load balancer) sits in front, so the gateway takes the client address from the right of X-Forwarded-For rather than the attacker-controlled left.

The orders team's route and policies, in its own namespace (labelled gateway-access: public):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: orders
  namespace: shop
spec:
  parentRefs:
    - name: public
      namespace: gateway-system
      sectionName: https
  hostnames:
    - api.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1/orders
      backendRefs:
        - name: orders-api
          port: 8080
      timeouts:
        request: 10s
    - matches:
        - path:
            type: PathPrefix
            value: /v2/orders
      backendRefs:
        - name: orders-api-v2
          port: 8080
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: orders-auth
  namespace: shop
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: orders
  jwt:
    providers:
      - name: corp-idp
        issuer: https://auth.example.com/
        audiences:
          - orders-api
        remoteJWKS:
          uri: https://auth.example.com/.well-known/jwks.json
        claimToHeaders:
          - claim: sub
            header: x-jwt-sub
  authorization:
    defaultAction: Deny
    rules:
      - name: require-api-scope
        action: Allow
        principal:
          jwt:
            provider: corp-idp
            scopes: ["orders.api"]
  cors:
    allowOrigins:
      - https://app.example.com
    allowMethods: [GET, POST]
    allowHeaders: [authorization, content-type]
    maxAge: 600s
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: orders-limits
  namespace: shop
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: orders
  rateLimit:
    global:
      rules:
        - clientSelectors:
            - headers:
                - name: x-jwt-sub
                  type: Distinct
          limit:
            requests: 120
            unit: Minute

Read it as one pipeline. The JWT provider validates issuer, audience, expiry and signature against the remote JWKS and copies the sub claim into x-jwt-sub. The authorization block denies any token lacking the orders.api scope (403). The rate limit keeps a separate counter for each distinct subject, 120 requests per minute. Global limits are shared across gateway replicas through Redis, which you enable once in the Envoy Gateway installation (config.envoyGateway.rateLimit.backend.type=Redis plus config.envoyGateway.rateLimit.backend.redis.url=<host>:6379 in its Helm values).

#Managed gateways

If you'd rather not run the proxy yourself, every cloud sells one. AWS API Gateway offers HTTP APIs (with built-in JWT authorizers) and REST APIs (with usage plans, API keys, throttling and request validation). Azure API Management expresses the same features as XML policies such as validate-jwt, rate-limit-by-key and quota-by-key, plus a developer portal. Google Apigee is a full API management platform with analytics, monetisation and a portal; Google Cloud API Gateway is a lighter option. The concepts in this lesson map directly onto all of them.

#Anti-patterns

• Business logic in the gateway. Pricing rules, order validation or response stitching scripted into gateway plugins are untestable, unversioned with the service, and owned by the wrong team. Keep the gateway to cross-cutting policy; put aggregation in a dedicated backend-for-frontend service.
• A single point of failure. One gateway instance in one zone means one crash takes down every API. Run at least two replicas across zones with a PodDisruptionBudget, autoscale them, and set timeouts so one slow backend can't tie up every gateway connection.
• The gateway as the only security layer. Backends reachable directly, or backends that trust any request carrying an x-user-id header, turn one bypass into total compromise.
• One giant shared config. If every change goes through one team's monolithic file, teams work around it. The Gateway API's Gateway/HTTPRoute split exists to prevent this.
• Logging secrets. Access logs full of bearer tokens and API keys are a credential leak waiting to happen.

#Practice

Two parts: one in code, one in configuration.

1. Write a per-key token bucket limiter in Python with allow(key, now) -> (allowed, retry_after_seconds). Using capacity 10 and 2 tokens per second, reproduce every row of the worked example table, and show that a second key has its own bucket.
2. Extend the Envoy Gateway configuration above for two new requirements: (a) a public, unauthenticated GET /v1/catalog route to the catalog-api Service (port 8080) in the same namespace, limited to 30 requests per minute per client IP; (b) send 10% of /v1/orders traffic to a canary Service orders-api-canary.

import math
from dataclasses import dataclass


@dataclass
class Bucket:
    tokens: float
    updated: float


class TokenBucketLimiter:
    def __init__(self, capacity: int, refill_per_sec: float) -> None:
        self.capacity = capacity
        self.rate = refill_per_sec
        self.buckets: dict[str, Bucket] = {}

    def allow(self, key: str, now: float) -> tuple[bool, float]:
        """Return (allowed, seconds until a token is available)."""
        b = self.buckets.get(key)
        if b is None:
            b = self.buckets[key] = Bucket(tokens=self.capacity, updated=now)
        b.tokens = min(self.capacity, b.tokens + (now - b.updated) * self.rate)
        b.updated = now
        if b.tokens >= 1:
            b.tokens -= 1
            return True, 0.0
        return False, (1 - b.tokens) / self.rate


if __name__ == "__main__":
    limiter = TokenBucketLimiter(capacity=10, refill_per_sec=2)

    burst = [limiter.allow("alice", 0.0)[0] for _ in range(12)]
    print(f"t=0.00 alice x12: {burst.count(True)} allowed, {burst.count(False)} rejected")

    print("t=1.00 alice x3: ", [limiter.allow("alice", 1.0)[0] for _ in range(3)])

    ok, wait = limiter.allow("alice", 1.25)
    print(f"t=1.25 alice:     allowed={ok}, retry in {wait:.2f}s -> Retry-After: {math.ceil(wait)}")

    print("t=1.25 bob:      ", limiter.allow("bob", 1.25)[0])

    limiter.allow("alice", 7.0)
    print(f"t=7.00 alice:     tokens left after 1 request = {limiter.buckets['alice'].tokens:.1f}")

$ python3 ratelimit.py
t=0.00 alice x12: 10 allowed, 2 rejected
t=1.00 alice x3:  [True, True, False]
t=1.25 alice:     allowed=False, retry in 0.25s -> Retry-After: 1
t=1.25 bob:       True
t=7.00 alice:     tokens left after 1 request = 9.0

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: catalog
  namespace: shop
spec:
  parentRefs:
    - name: public
      namespace: gateway-system
      sectionName: https
  hostnames:
    - api.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1/catalog
          method: GET
      backendRefs:
        - name: catalog-api
          port: 8080
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: catalog-limits
  namespace: shop
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: catalog
  rateLimit:
    global:
      rules:
        - clientSelectors:
            - sourceCIDR:
                type: Distinct
                value: 0.0.0.0/0
          limit:
            requests: 30
            unit: Minute

    - matches:
        - path:
            type: PathPrefix
            value: /v1/orders
      backendRefs:
        - name: orders-api
          port: 8080
          weight: 90
        - name: orders-api-canary
          port: 8080
          weight: 10
      timeouts:
        request: 10s

#Key takeaways

• A load balancer picks a healthy backend, a service mesh secures service-to-service calls, and an API gateway applies API-level policy (identity, routes, limits) to traffic entering from outside.
• Validate JWTs at the edge against a cached JWKS: pin the algorithms, match kid, verify the signature, and check iss, aud and exp; use client credentials instead of long-lived keys for machines.
• The gateway authenticates; the backend still authorises every object access, and it must be reachable only through the gateway.
• A token bucket allows bursts up to its capacity and a long-run rate equal to its refill; limit per key or subject where you have identity, per IP for anonymous and login endpoints.
• Enforce size limits, strict CORS allowlists and a TLS policy at the gateway, and log subjects, never tokens.
• Keep business logic out of the gateway, run it redundantly across zones, and let the Gateway API split ownership between platform and app teams.