Service mesh & mTLS

Your edge is protected by TLS, a gateway and a WAF. Inside the cluster, though, services usually call each other over plain HTTP, and anything that can reach a pod's IP can talk to it. A service mesh moves encryption, authentication and authorization for that internal traffic out of your application code and into infrastructure, so every call between services is encrypted, tied to a verified workload identity and checked against policy. In this optional lesson you'll learn how meshes work (sidecar and sidecar-less), what mutual TLS and SPIFFE identities add, how to enforce them with Istio's PeerAuthentication and AuthorizationPolicy, what Linkerd looks like, which traffic features come along, and the costs that should make you hesitate before installing one.

#The problem with east-west traffic

Traffic entering the cluster from outside is north-south; traffic between services inside it is east-west. Teams tend to secure the first and trust the second, which is the old perimeter model: once an attacker has code execution in any pod, the internal network is theirs. They can sniff credentials from plaintext calls on the same node, call internal admin endpoints, and impersonate one service to another, because nothing checks who is calling.

Zero trust replaces "trusted because it's on the network" with three checks on every connection:

1. Authentication. Who is calling, proven cryptographically, not inferred from an IP address.
2. Encryption. Nobody on the path can read or modify the traffic.
3. Authorization. This caller may perform this operation on this service.

Kubernetes NetworkPolicies (covered in the Kubernetes security lesson, later in this roadmap) are a good start, but they work on pod labels and IP addresses at layers 3 and 4. They don't encrypt anything, they can't tell GET /orders from DELETE /orders, and they trust that an IP belongs to the pod it was assigned to. You could add mutual TLS to every service yourself, but then every team, in every language, has to issue certificates, rotate them, validate peers and write authorization checks. A mesh does it once, consistently, for all of them.

#How a mesh works

Every mesh has two parts:

• A data plane of proxies that sit in the path of every connection. Applications send ordinary requests; the proxies add TLS, check policy, retry, time out and record metrics.
• A control plane that tells the proxies what to do. It watches Kubernetes for Services and policies, pushes configuration to the proxies, and acts as a certificate authority that issues each workload its identity.

Where the proxies run is the big architectural choice.

Sidecar mode. A proxy container is injected into every pod (Envoy for Istio, the Rust-based linkerd2-proxy for Linkerd), and the pod's iptables rules redirect all inbound and outbound traffic through it. Each pod gets full layer-7 features and its own isolated proxy. The cost is a proxy per pod: CPU and memory multiplied by your replica count, an extra hop on both sides of every call, and a restart of every pod whenever the proxy is upgraded.

Sidecar-less (ambient) mode. Istio's ambient mode splits the work. A per-node agent called ztunnel handles layer 4 for every pod on the node: mTLS, identity and connection-level authorization. Pods enrolled in the mesh need no change and no restart. When a service needs layer-7 features (HTTP-aware authorization, retries, traffic splitting), you deploy a waypoint proxy, a shared Envoy for a namespace or service. Cilium's service mesh takes a similar approach with eBPF and a per-node Envoy.

#Workload identity and mutual TLS

In ordinary TLS, as in the networking lesson, only the server proves its identity. In mutual TLS (mTLS) the client presents a certificate too, so each side knows exactly who the other is before any application data flows.

What should the certificate name? Not an IP address, which changes every time a pod restarts, but the workload. Meshes use SPIFFE (Secure Production Identity Framework For Everyone), a standard for workload identity. A SPIFFE ID is a URI, and Istio derives it from the pod's namespace and ServiceAccount:

spiffe://cluster.local/ns/orders/sa/orders-api
         └ trust domain ┘ └ namespace ┘ └ ServiceAccount ┘

The ID is carried in the URI Subject Alternative Name of a short-lived X.509 certificate, called an SVID. Istio's control plane (istiod) signs one for each workload when the proxy starts, valid for 24 hours by default, and the proxy renews it automatically well before expiry. There's nothing to store, distribute or rotate by hand, and a stolen certificate is useless within a day. This is also why one ServiceAccount per workload matters: two Deployments sharing a ServiceAccount share an identity, and no policy can tell them apart.

Here is one call from orders-api to payments-api in sidecar mode. Watch where the plaintext is: only on localhost inside each pod, never on the network.

In ambient mode the shape is the same, except the two ztunnels on the client's and server's nodes play the proxy roles, tunnelling the connection over mTLS on port 15008.

#Enforcing mTLS and authorization with Istio

By default Istio runs in PERMISSIVE mode: meshed clients automatically use mTLS with meshed servers, but servers still accept plaintext, so workloads outside the mesh keep working while you migrate. Permissive is a migration state, not a security control, because an attacker's pod simply connects in plaintext.

A PeerAuthentication in STRICT mode makes servers refuse anything that isn't mTLS. Put it in a namespace to cover that namespace, or in the root namespace (istio-system) to cover the whole mesh:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system      # root namespace: applies mesh-wide
spec:
  mtls:
    mode: STRICT

Before switching, confirm nothing still calls in plaintext: Istio's request metrics carry a connection_security_policy label (mutual_tls or none), and the Kiali dashboard shows the same per edge. After switching, a client outside the mesh gets its connection reset:

$ kubectl exec -n legacy deploy/batch-job -- curl -sS http://payments-api.payments/health
curl: (56) Recv failure: Connection reset by peer
command terminated with exit code 56

mTLS proves who is calling; AuthorizationPolicy decides what they may do. Start each namespace with an allow-nothing policy (an ALLOW policy with no rules matches nothing, so everything is denied), then allow specific callers by principal, which is the SPIFFE ID without the spiffe:// prefix:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-nothing
  namespace: payments
spec: {}
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: payments-api
  namespace: payments
spec:
  selector:
    matchLabels:
      app: payments-api
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/orders/sa/orders-api"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/charges", "/charges/*"]
    - from:
        - source:
            principals: ["cluster.local/ns/monitoring/sa/prometheus"]
      to:
        - operation:
            methods: ["GET"]
            paths: ["/metrics"]

Any other caller, or orders-api trying DELETE /charges/123, is refused by the payments proxy:

$ kubectl exec -n orders deploy/orders-api -c api -- curl -s -w ' %{http_code}\n' -X DELETE http://payments-api.payments/charges/123
RBAC: access denied 403

Policies are evaluated CUSTOM, then DENY, then ALLOW: a matching DENY always wins, and once any ALLOW policy selects a workload, requests that match none of its rules are rejected. In ambient mode, ztunnel enforces rules that use only layer-4 attributes (principals, namespaces, ports); rules on methods and paths need a waypoint proxy, attached with targetRefs instead of selector.

#Linkerd in brief

Linkerd is the other widely used mesh. It is deliberately smaller: sidecar-only, with its own lightweight Rust proxy rather than Envoy, and very little to configure. mTLS is on automatically for all TCP traffic between meshed pods the moment they're injected. Identities are derived from ServiceAccounts in a DNS-like form such as orders-api.orders.serviceaccount.identity.linkerd.cluster.local.

Injection and a default policy are namespace annotations; finer-grained rules use a Server (a port on a set of pods) plus an AuthorizationPolicy naming who may connect:

apiVersion: v1
kind: Namespace
metadata:
  name: payments
  annotations:
    linkerd.io/inject: enabled
    config.linkerd.io/default-inbound-policy: all-authenticated   # meshed callers only
---
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata:
  name: payments-api-http
  namespace: payments
spec:
  podSelector:
    matchLabels: { app: payments-api }
  port: http
  proxyProtocol: HTTP/1
---
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata:
  name: orders-api
  namespace: payments
spec:
  identities:
    - "orders-api.orders.serviceaccount.identity.linkerd.cluster.local"
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata:
  name: payments-api-from-orders
  namespace: payments
spec:
  targetRef:
    group: policy.linkerd.io
    kind: Server
    name: payments-api-http
  requiredAuthenticationRefs:
    - name: orders-api
      kind: MeshTLSAuthentication
      group: policy.linkerd.io

Once a Server exists, traffic to that port is denied unless an AuthorizationPolicy allows it. Note the packaging: the open-source project publishes edge releases, while stable releases come from its main sponsor, Buoyant, under its own terms. Check that fits your organisation before you standardise on it.

#Traffic management

Because the proxies understand HTTP, meshes also offer reliability features without code changes. In Istio, a DestinationRule defines subsets of a service and a VirtualService routes between them, here sending 10% of traffic to a canary and adding a timeout and retries:

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: catalog
  namespace: shop
spec:
  host: catalog.shop.svc.cluster.local
  subsets:
    - name: stable
      labels: { version: v1 }
    - name: canary
      labels: { version: v2 }
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: catalog
  namespace: shop
spec:
  hosts: ["catalog.shop.svc.cluster.local"]
  http:
    - timeout: 2s
      retries:
        attempts: 2
        perTryTimeout: 500ms
        retryOn: connect-failure,reset,503
      route:
        - destination: { host: catalog.shop.svc.cluster.local, subset: stable }
          weight: 90
        - destination: { host: catalog.shop.svc.cluster.local, subset: canary }
          weight: 10

Both Istio and Linkerd also accept the Kubernetes Gateway API's HTTPRoute for in-mesh routing, with weighted backendRefs for canaries, which keeps your routing config portable between meshes.

Two pitfalls. Retries must be safe to repeat: retrying a POST /charges that timed out after the payment went through charges the customer twice, so retry idempotent calls only, or make the endpoint idempotent with request keys. And retries multiply: if three layers of services each retry three times, one failing request at the bottom becomes up to 27 at peak, exactly when the system is struggling. Keep attempts low, and set timeouts so an outer call's budget covers its inner calls.

The other dividend is telemetry: every proxy records request rate, error rate and latency per caller and callee, so you get a consistent service graph and golden metrics for every service, including ones nobody instrumented.

#Costs, and when not to adopt one

A mesh is a distributed system you now operate, in the path of every request:

• Resources and latency. Proxies use CPU and memory (per pod for sidecars, per node for ambient) and add a small but real latency to each hop.
• Operations. The control plane and the proxies need regular upgrades, which in sidecar mode means restarting every workload. Dozens of CRDs and a new CA become part of your platform.
• New failure modes. A bad policy can block traffic cluster-wide in seconds. If the control plane is down, existing proxies keep working on their current config and certificates, but new pods can't get certificates and changes don't propagate.
• Debugging. "Connection reset" can now mean the app, the client proxy, the server proxy or a policy. Your team needs to learn the proxy's logs and diagnostic commands (istioctl analyze, istioctl proxy-config, linkerd viz).

So don't adopt one by default. A monolith or a handful of services owned by one team gains little. If what you need is encryption in transit between nodes, your CNI may already offer it (Cilium and Calico both support WireGuard node-to-node encryption) at far lower cost. If you need to restrict who talks to whom at the network level, NetworkPolicy may be enough.

A mesh earns its keep when you have many services owned by several teams, a requirement (regulatory or contractual) for encryption and authenticated service identity on every internal call, a need for layer-7 authorization between services, or a desire for uniform telemetry without touching every codebase. If you adopt one, do it in steps: install, enrol namespaces in permissive mode, watch the metrics until all traffic is mutual TLS, switch to STRICT, then add allow-nothing and specific authorization policies one namespace at a time.

#Practice

On a kind cluster, install Istio in sidecar mode and secure one service:

1. Install Istio: download istioctl (curl -L https://istio.io/downloadIstio | sh -, then add its bin directory to your PATH) and run istioctl install --set profile=minimal -y.
2. Create a namespace shop with sidecar injection, containing a backend Deployment (traefik/whoami:v1.12.0 on port 8080, ServiceAccount backend, Service on port 80) and two client Deployments, frontend and other, each with its own ServiceAccount, running curlimages/curl:8.22.0 with sleep infinity. Create a namespace legacy, without injection, with one more client.
3. Call backend from all three clients and look at the headers whoami prints.
4. Enforce STRICT mTLS in shop and show the legacy client is refused.
5. Allow only frontend, and only GET, to reach backend; show other and a POST from frontend get 403.

# mesh-lab.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: shop
  labels:
    istio-injection: enabled
---
apiVersion: v1
kind: Namespace
metadata:
  name: legacy
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: backend, namespace: shop }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: backend, namespace: shop }
spec:
  replicas: 1
  selector:
    matchLabels: { app: backend }
  template:
    metadata:
      labels: { app: backend }
    spec:
      serviceAccountName: backend
      containers:
        - name: whoami
          image: traefik/whoami:v1.12.0
          env:
            - { name: WHOAMI_PORT_NUMBER, value: "8080" }
          ports:
            - { name: http, containerPort: 8080 }
---
apiVersion: v1
kind: Service
metadata: { name: backend, namespace: shop }
spec:
  selector: { app: backend }
  ports:
    - { name: http, port: 80, targetPort: http }
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: frontend, namespace: shop }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: frontend, namespace: shop }
spec:
  selector:
    matchLabels: { app: frontend }
  template:
    metadata:
      labels: { app: frontend }
    spec:
      serviceAccountName: frontend
      containers:
        - name: curl
          image: curlimages/curl:8.22.0
          command: ["sleep", "infinity"]
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: other, namespace: shop }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: other, namespace: shop }
spec:
  selector:
    matchLabels: { app: other }
  template:
    metadata:
      labels: { app: other }
    spec:
      serviceAccountName: other
      containers:
        - name: curl
          image: curlimages/curl:8.22.0
          command: ["sleep", "infinity"]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: legacy, namespace: legacy }
spec:
  selector:
    matchLabels: { app: legacy }
  template:
    metadata:
      labels: { app: legacy }
    spec:
      containers:
        - name: curl
          image: curlimages/curl:8.22.0
          command: ["sleep", "infinity"]

kubectl apply -f mesh-lab.yaml
kubectl get pods -n shop      # READY 2/2: app container + istio-proxy
kubectl exec -n shop deploy/frontend -c curl -- curl -s http://backend/ | grep -i -e hostname -e x-forwarded-client-cert
kubectl exec -n legacy deploy/legacy -- curl -s -o /dev/null -w '%{http_code}\n' http://backend.shop/

Hostname: backend-7b9d6c5f4-kx8qz
X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/shop/sa/backend;Hash=4f1c...;Subject="";URI=spiffe://cluster.local/ns/shop/sa/frontend
200

# mesh-policy.yaml
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: { name: default, namespace: shop }
spec:
  mtls:
    mode: STRICT
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: { name: allow-nothing, namespace: shop }
spec: {}
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: { name: backend, namespace: shop }
spec:
  selector:
    matchLabels: { app: backend }
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/shop/sa/frontend"]
      to:
        - operation:
            methods: ["GET"]

kubectl apply -f mesh-policy.yaml
kubectl exec -n legacy deploy/legacy -- curl -sS http://backend.shop/
kubectl exec -n shop deploy/other -c curl -- curl -s -w ' %{http_code}\n' http://backend/
kubectl exec -n shop deploy/frontend -c curl -- curl -s -w ' %{http_code}\n' -X POST http://backend/
kubectl exec -n shop deploy/frontend -c curl -- curl -s -o /dev/null -w '%{http_code}\n' http://backend/

curl: (56) Recv failure: Connection reset by peer
command terminated with exit code 56
RBAC: access denied 403
RBAC: access denied 403
200

#Key takeaways

• A service mesh puts proxies in the path of service-to-service traffic, configured by a control plane that also issues workload certificates; sidecar mode runs a proxy per pod, ambient mode a per-node ztunnel plus optional L7 waypoints.
• mTLS gives every connection encryption and a cryptographic caller identity; SPIFFE IDs derived from namespace and ServiceAccount name the workload, not its IP.
• Permissive mode is for migration only: enforce STRICT PeerAuthentication, then default-deny with an allow-nothing AuthorizationPolicy and allow specific principals, methods and paths.
• Workload identity is only as strong as control over who can run pods as a ServiceAccount, so mesh policy builds on RBAC, Pod Security and one ServiceAccount per workload.
• Retries, timeouts and weighted canary routing come with the mesh; retry only idempotent calls and keep attempts low to avoid retry storms.
• A mesh costs resources, latency and operational effort; adopt one for many services, multiple teams or compliance needs, and consider CNI encryption or NetworkPolicy when they suffice.