# Service mesh & mTLS > Explain how a service mesh secures service-to-service traffic, enforce mTLS and identity-based authorization with Istio, and judge when one is worth it. DevSecOps Engineer roadmap · Run the platform · 20 min read · https://levelcode.org/roadmaps/devsecops/service-mesh Your edge is protected by TLS, a gateway and a WAF. Inside the cluster, though, services usually call each other over plain HTTP, and anything that can reach a pod's IP can talk to it. A **service mesh** moves encryption, authentication and authorization for that internal traffic out of your application code and into infrastructure, so every call between services is encrypted, tied to a verified workload identity and checked against policy. In this optional lesson you'll learn how meshes work (sidecar and sidecar-less), what mutual TLS and SPIFFE identities add, how to enforce them with Istio's `PeerAuthentication` and `AuthorizationPolicy`, what Linkerd looks like, which traffic features come along, and the costs that should make you hesitate before installing one. ## The problem with east-west traffic Traffic entering the cluster from outside is **north-south**; traffic between services inside it is **east-west**. Teams tend to secure the first and trust the second, which is the old perimeter model: once an attacker has code execution in any pod, the internal network is theirs. They can sniff credentials from plaintext calls on the same node, call internal admin endpoints, and impersonate one service to another, because nothing checks who is calling. **Zero trust** replaces "trusted because it's on the network" with three checks on every connection: 1. **Authentication.** Who is calling, proven cryptographically, not inferred from an IP address. 2. **Encryption.** Nobody on the path can read or modify the traffic. 3. **Authorization.** This caller may perform this operation on this service. Kubernetes NetworkPolicies (covered in the Kubernetes security lesson, later in this roadmap) are a good start, but they work on pod labels and IP addresses at layers 3 and 4. They don't encrypt anything, they can't tell `GET /orders` from `DELETE /orders`, and they trust that an IP belongs to the pod it was assigned to. You could add mutual TLS to every service yourself, but then every team, in every language, has to issue certificates, rotate them, validate peers and write authorization checks. A mesh does it once, consistently, for all of them. ## How a mesh works Every mesh has two parts: - A **data plane** of proxies that sit in the path of every connection. Applications send ordinary requests; the proxies add TLS, check policy, retry, time out and record metrics. - A **control plane** that tells the proxies what to do. It watches Kubernetes for Services and policies, pushes configuration to the proxies, and acts as a certificate authority that issues each workload its identity. Where the proxies run is the big architectural choice. **Sidecar mode.** A proxy container is injected into every pod (Envoy for Istio, the Rust-based `linkerd2-proxy` for Linkerd), and the pod's iptables rules redirect all inbound and outbound traffic through it. Each pod gets full layer-7 features and its own isolated proxy. The cost is a proxy per pod: CPU and memory multiplied by your replica count, an extra hop on both sides of every call, and a restart of every pod whenever the proxy is upgraded. **Sidecar-less (ambient) mode.** Istio's **ambient mode** splits the work. A per-node agent called **ztunnel** handles layer 4 for every pod on the node: mTLS, identity and connection-level authorization. Pods enrolled in the mesh need no change and no restart. When a service needs layer-7 features (HTTP-aware authorization, retries, traffic splitting), you deploy a **waypoint proxy**, a shared Envoy for a namespace or service. Cilium's service mesh takes a similar approach with eBPF and a per-node Envoy. | | Sidecar | Ambient / sidecar-less | |---|---|---| | Proxy placement | One per pod | One per node (L4), optional shared waypoints (L7) | | Resource cost | Grows with pod count | Grows with node count, plus waypoints | | Joining the mesh | Inject and restart pods | Label the namespace; no restart | | L7 features | Everywhere, automatically | Only where you deploy a waypoint | | Blast radius of a proxy bug | One pod | Every pod on a node (for ztunnel) | ## Workload identity and mutual TLS In ordinary TLS, as in the networking lesson, only the server proves its identity. In **mutual TLS (mTLS)** the client presents a certificate too, so each side knows exactly who the other is before any application data flows. What should the certificate name? Not an IP address, which changes every time a pod restarts, but the *workload*. Meshes use **SPIFFE** (Secure Production Identity Framework For Everyone), a standard for workload identity. A SPIFFE ID is a URI, and Istio derives it from the pod's namespace and ServiceAccount: ```text spiffe://cluster.local/ns/orders/sa/orders-api └ trust domain ┘ └ namespace ┘ └ ServiceAccount ┘ ``` The ID is carried in the URI Subject Alternative Name of a short-lived X.509 certificate, called an **SVID**. Istio's control plane (istiod) signs one for each workload when the proxy starts, valid for 24 hours by default, and the proxy renews it automatically well before expiry. There's nothing to store, distribute or rotate by hand, and a stolen certificate is useless within a day. This is also why one ServiceAccount per workload matters: two Deployments sharing a ServiceAccount share an identity, and no policy can tell them apart. Here is one call from `orders-api` to `payments-api` in sidecar mode. Watch where the plaintext is: only on `localhost` inside each pod, never on the network. ```sequence title: An mTLS call through two sidecar proxies participants: Orders app, Orders proxy, Payments proxy, Payments app, istiod istiod --> Orders proxy: SVID for ns/orders/sa/orders-api > At startup each proxy generates a private key, sends a certificate signing request with its pod's ServiceAccount token, and receives a 24-hour certificate. Keys never leave the pod. Orders app -> Orders proxy: POST http://payments-api/charges > The app sends plain HTTP to the Service name. iptables redirects the outbound connection to the local proxy; the app needs no TLS code. Orders proxy -> Payments proxy: mTLS handshake: both send SVIDs > The proxy opens a TLS 1.3 connection to a payments pod. The server asks for a client certificate and the proxy presents its own SVID. The client also checks that the server's certificate names the identity expected for payments-api. Payments proxy -> Payments proxy: Verify chain, read SPIFFE ID > The certificate must chain to the mesh CA. The peer's identity, spiffe://cluster.local/ns/orders/sa/orders-api, now comes from cryptography, not from an IP address. Payments proxy -> Payments proxy: AuthorizationPolicy: allow? > Is this principal allowed to POST to /charges? If not, the proxy answers 403 and the app never sees the request. Payments proxy -> Payments app: POST /charges (localhost) > Forwarded to the app inside the same pod, with an X-Forwarded-Client-Cert header naming the caller. Payments app --> Payments proxy: 201 Created Payments proxy --> Orders proxy: 201 Created (encrypted) Orders proxy --> Orders app: 201 Created ``` In ambient mode the shape is the same, except the two ztunnels on the client's and server's nodes play the proxy roles, tunnelling the connection over mTLS on port 15008. ## Enforcing mTLS and authorization with Istio By default Istio runs in **PERMISSIVE** mode: meshed clients automatically use mTLS with meshed servers, but servers still accept plaintext, so workloads outside the mesh keep working while you migrate. Permissive is a migration state, not a security control, because an attacker's pod simply connects in plaintext. A **PeerAuthentication** in **STRICT** mode makes servers refuse anything that isn't mTLS. Put it in a namespace to cover that namespace, or in the root namespace (`istio-system`) to cover the whole mesh: ```yaml apiVersion: security.istio.io/v1 kind: PeerAuthentication metadata: name: default namespace: istio-system # root namespace: applies mesh-wide spec: mtls: mode: STRICT ``` Before switching, confirm nothing still calls in plaintext: Istio's request metrics carry a `connection_security_policy` label (`mutual_tls` or `none`), and the Kiali dashboard shows the same per edge. After switching, a client outside the mesh gets its connection reset: ```text $ kubectl exec -n legacy deploy/batch-job -- curl -sS http://payments-api.payments/health curl: (56) Recv failure: Connection reset by peer command terminated with exit code 56 ``` mTLS proves who is calling; **AuthorizationPolicy** decides what they may do. Start each namespace with an allow-nothing policy (an `ALLOW` policy with no rules matches nothing, so everything is denied), then allow specific callers by principal, which is the SPIFFE ID without the `spiffe://` prefix: ```yaml apiVersion: security.istio.io/v1 kind: AuthorizationPolicy metadata: name: allow-nothing namespace: payments spec: {} --- apiVersion: security.istio.io/v1 kind: AuthorizationPolicy metadata: name: payments-api namespace: payments spec: selector: matchLabels: app: payments-api action: ALLOW rules: - from: - source: principals: ["cluster.local/ns/orders/sa/orders-api"] to: - operation: methods: ["POST"] paths: ["/charges", "/charges/*"] - from: - source: principals: ["cluster.local/ns/monitoring/sa/prometheus"] to: - operation: methods: ["GET"] paths: ["/metrics"] ``` Any other caller, or `orders-api` trying `DELETE /charges/123`, is refused by the payments proxy: ```text $ kubectl exec -n orders deploy/orders-api -c api -- curl -s -w ' %{http_code}\n' -X DELETE http://payments-api.payments/charges/123 RBAC: access denied 403 ``` Policies are evaluated `CUSTOM`, then `DENY`, then `ALLOW`: a matching `DENY` always wins, and once any `ALLOW` policy selects a workload, requests that match none of its rules are rejected. In ambient mode, ztunnel enforces rules that use only layer-4 attributes (principals, namespaces, ports); rules on methods and paths need a waypoint proxy, attached with `targetRefs` instead of `selector`. > [!WARNING] > Identity-based policy is only as good as the identities. Anyone who can create pods with the `orders-api` ServiceAccount in `orders` can call payments as `orders-api`. RBAC on pod creation, Pod Security and one ServiceAccount per workload, covered in the Kubernetes security lesson, are what make the principal mean something. ## Linkerd in brief Linkerd is the other widely used mesh. It is deliberately smaller: sidecar-only, with its own lightweight Rust proxy rather than Envoy, and very little to configure. mTLS is on automatically for all TCP traffic between meshed pods the moment they're injected. Identities are derived from ServiceAccounts in a DNS-like form such as `orders-api.orders.serviceaccount.identity.linkerd.cluster.local`. Injection and a default policy are namespace annotations; finer-grained rules use a `Server` (a port on a set of pods) plus an `AuthorizationPolicy` naming who may connect: ```yaml apiVersion: v1 kind: Namespace metadata: name: payments annotations: linkerd.io/inject: enabled config.linkerd.io/default-inbound-policy: all-authenticated # meshed callers only --- apiVersion: policy.linkerd.io/v1beta3 kind: Server metadata: name: payments-api-http namespace: payments spec: podSelector: matchLabels: { app: payments-api } port: http proxyProtocol: HTTP/1 --- apiVersion: policy.linkerd.io/v1alpha1 kind: MeshTLSAuthentication metadata: name: orders-api namespace: payments spec: identities: - "orders-api.orders.serviceaccount.identity.linkerd.cluster.local" --- apiVersion: policy.linkerd.io/v1alpha1 kind: AuthorizationPolicy metadata: name: payments-api-from-orders namespace: payments spec: targetRef: group: policy.linkerd.io kind: Server name: payments-api-http requiredAuthenticationRefs: - name: orders-api kind: MeshTLSAuthentication group: policy.linkerd.io ``` Once a `Server` exists, traffic to that port is denied unless an `AuthorizationPolicy` allows it. Note the packaging: the open-source project publishes **edge** releases, while stable releases come from its main sponsor, Buoyant, under its own terms. Check that fits your organisation before you standardise on it. | | Istio | Linkerd | |---|---|---| | Data plane | Envoy sidecars, or ambient (ztunnel + waypoints) | Rust micro-proxy sidecars | | mTLS | Automatic; STRICT via PeerAuthentication | Automatic for all meshed TCP | | Feature set | Very broad: L7 policy, traffic shifting, egress control, multi-cluster | Focused core; fewer knobs | | Operational weight | Heavier; many CRDs | Lighter; simpler to run | ## Traffic management Because the proxies understand HTTP, meshes also offer reliability features without code changes. In Istio, a `DestinationRule` defines subsets of a service and a `VirtualService` routes between them, here sending 10% of traffic to a canary and adding a timeout and retries: ```yaml apiVersion: networking.istio.io/v1 kind: DestinationRule metadata: name: catalog namespace: shop spec: host: catalog.shop.svc.cluster.local subsets: - name: stable labels: { version: v1 } - name: canary labels: { version: v2 } --- apiVersion: networking.istio.io/v1 kind: VirtualService metadata: name: catalog namespace: shop spec: hosts: ["catalog.shop.svc.cluster.local"] http: - timeout: 2s retries: attempts: 2 perTryTimeout: 500ms retryOn: connect-failure,reset,503 route: - destination: { host: catalog.shop.svc.cluster.local, subset: stable } weight: 90 - destination: { host: catalog.shop.svc.cluster.local, subset: canary } weight: 10 ``` Both Istio and Linkerd also accept the Kubernetes Gateway API's `HTTPRoute` for in-mesh routing, with weighted `backendRefs` for canaries, which keeps your routing config portable between meshes. Two pitfalls. **Retries must be safe to repeat**: retrying a `POST /charges` that timed out after the payment went through charges the customer twice, so retry idempotent calls only, or make the endpoint idempotent with request keys. And **retries multiply**: if three layers of services each retry three times, one failing request at the bottom becomes up to 27 at peak, exactly when the system is struggling. Keep attempts low, and set timeouts so an outer call's budget covers its inner calls. The other dividend is **telemetry**: every proxy records request rate, error rate and latency per caller and callee, so you get a consistent service graph and golden metrics for every service, including ones nobody instrumented. ## Costs, and when not to adopt one A mesh is a distributed system you now operate, in the path of every request: - **Resources and latency.** Proxies use CPU and memory (per pod for sidecars, per node for ambient) and add a small but real latency to each hop. - **Operations.** The control plane and the proxies need regular upgrades, which in sidecar mode means restarting every workload. Dozens of CRDs and a new CA become part of your platform. - **New failure modes.** A bad policy can block traffic cluster-wide in seconds. If the control plane is down, existing proxies keep working on their current config and certificates, but new pods can't get certificates and changes don't propagate. - **Debugging.** "Connection reset" can now mean the app, the client proxy, the server proxy or a policy. Your team needs to learn the proxy's logs and diagnostic commands (`istioctl analyze`, `istioctl proxy-config`, `linkerd viz`). So don't adopt one by default. A monolith or a handful of services owned by one team gains little. If what you need is encryption in transit between nodes, your CNI may already offer it (Cilium and Calico both support WireGuard node-to-node encryption) at far lower cost. If you need to restrict who talks to whom at the network level, NetworkPolicy may be enough. A mesh earns its keep when you have many services owned by several teams, a requirement (regulatory or contractual) for encryption and authenticated service identity on every internal call, a need for layer-7 authorization between services, or a desire for uniform telemetry without touching every codebase. If you adopt one, do it in steps: install, enrol namespaces in permissive mode, watch the metrics until all traffic is mutual TLS, switch to STRICT, then add allow-nothing and specific authorization policies one namespace at a time. ## Practice On a kind cluster, install Istio in sidecar mode and secure one service: 1. Install Istio: download `istioctl` (`curl -L https://istio.io/downloadIstio | sh -`, then add its `bin` directory to your `PATH`) and run `istioctl install --set profile=minimal -y`. 2. Create a namespace `shop` with sidecar injection, containing a `backend` Deployment (`traefik/whoami:v1.12.0` on port 8080, ServiceAccount `backend`, Service on port 80) and two client Deployments, `frontend` and `other`, each with its own ServiceAccount, running `curlimages/curl:8.22.0` with `sleep infinity`. Create a namespace `legacy`, without injection, with one more client. 3. Call `backend` from all three clients and look at the headers whoami prints. 4. Enforce STRICT mTLS in `shop` and show the `legacy` client is refused. 5. Allow only `frontend`, and only `GET`, to reach `backend`; show `other` and a `POST` from `frontend` get 403. **Solution** ```yaml # mesh-lab.yaml apiVersion: v1 kind: Namespace metadata: name: shop labels: istio-injection: enabled --- apiVersion: v1 kind: Namespace metadata: name: legacy --- apiVersion: v1 kind: ServiceAccount metadata: { name: backend, namespace: shop } --- apiVersion: apps/v1 kind: Deployment metadata: { name: backend, namespace: shop } spec: replicas: 1 selector: matchLabels: { app: backend } template: metadata: labels: { app: backend } spec: serviceAccountName: backend containers: - name: whoami image: traefik/whoami:v1.12.0 env: - { name: WHOAMI_PORT_NUMBER, value: "8080" } ports: - { name: http, containerPort: 8080 } --- apiVersion: v1 kind: Service metadata: { name: backend, namespace: shop } spec: selector: { app: backend } ports: - { name: http, port: 80, targetPort: http } --- apiVersion: v1 kind: ServiceAccount metadata: { name: frontend, namespace: shop } --- apiVersion: apps/v1 kind: Deployment metadata: { name: frontend, namespace: shop } spec: selector: matchLabels: { app: frontend } template: metadata: labels: { app: frontend } spec: serviceAccountName: frontend containers: - name: curl image: curlimages/curl:8.22.0 command: ["sleep", "infinity"] --- apiVersion: v1 kind: ServiceAccount metadata: { name: other, namespace: shop } --- apiVersion: apps/v1 kind: Deployment metadata: { name: other, namespace: shop } spec: selector: matchLabels: { app: other } template: metadata: labels: { app: other } spec: serviceAccountName: other containers: - name: curl image: curlimages/curl:8.22.0 command: ["sleep", "infinity"] --- apiVersion: apps/v1 kind: Deployment metadata: { name: legacy, namespace: legacy } spec: selector: matchLabels: { app: legacy } template: metadata: labels: { app: legacy } spec: containers: - name: curl image: curlimages/curl:8.22.0 command: ["sleep", "infinity"] ``` ```bash kubectl apply -f mesh-lab.yaml kubectl get pods -n shop # READY 2/2: app container + istio-proxy kubectl exec -n shop deploy/frontend -c curl -- curl -s http://backend/ | grep -i -e hostname -e x-forwarded-client-cert kubectl exec -n legacy deploy/legacy -- curl -s -o /dev/null -w '%{http_code}\n' http://backend.shop/ ``` ```text Hostname: backend-7b9d6c5f4-kx8qz X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/shop/sa/backend;Hash=4f1c...;Subject="";URI=spiffe://cluster.local/ns/shop/sa/frontend 200 ``` The frontend call already used mTLS (auto mTLS in permissive mode), and the backend's proxy told the app who called. The legacy call worked in plaintext. Now lock it down: ```yaml # mesh-policy.yaml apiVersion: security.istio.io/v1 kind: PeerAuthentication metadata: { name: default, namespace: shop } spec: mtls: mode: STRICT --- apiVersion: security.istio.io/v1 kind: AuthorizationPolicy metadata: { name: allow-nothing, namespace: shop } spec: {} --- apiVersion: security.istio.io/v1 kind: AuthorizationPolicy metadata: { name: backend, namespace: shop } spec: selector: matchLabels: { app: backend } action: ALLOW rules: - from: - source: principals: ["cluster.local/ns/shop/sa/frontend"] to: - operation: methods: ["GET"] ``` ```bash kubectl apply -f mesh-policy.yaml kubectl exec -n legacy deploy/legacy -- curl -sS http://backend.shop/ kubectl exec -n shop deploy/other -c curl -- curl -s -w ' %{http_code}\n' http://backend/ kubectl exec -n shop deploy/frontend -c curl -- curl -s -w ' %{http_code}\n' -X POST http://backend/ kubectl exec -n shop deploy/frontend -c curl -- curl -s -o /dev/null -w '%{http_code}\n' http://backend/ ``` ```text curl: (56) Recv failure: Connection reset by peer command terminated with exit code 56 RBAC: access denied 403 RBAC: access denied 403 200 ``` The legacy pod has no certificate, so the STRICT server proxy resets it. `other` has a valid mesh identity, but the policy doesn't allow it. `frontend` may `GET` but not `POST`. Policies can take a few seconds to reach the proxies, so retry once if the first result surprises you. Clean up with `kind delete cluster`. ## Key takeaways - A service mesh puts proxies in the path of service-to-service traffic, configured by a control plane that also issues workload certificates; sidecar mode runs a proxy per pod, ambient mode a per-node ztunnel plus optional L7 waypoints. - mTLS gives every connection encryption and a cryptographic caller identity; SPIFFE IDs derived from namespace and ServiceAccount name the workload, not its IP. - Permissive mode is for migration only: enforce `STRICT` PeerAuthentication, then default-deny with an allow-nothing AuthorizationPolicy and allow specific principals, methods and paths. - Workload identity is only as strong as control over who can run pods as a ServiceAccount, so mesh policy builds on RBAC, Pod Security and one ServiceAccount per workload. - Retries, timeouts and weighted canary routing come with the mesh; retry only idempotent calls and keep attempts low to avoid retry storms. - A mesh costs resources, latency and operational effort; adopt one for many services, multiple teams or compliance needs, and consider CNI encryption or NetworkPolicy when they suffice. ## Quiz 1. A namespace's PeerAuthentication is in PERMISSIVE mode. What happens when a pod outside the mesh sends a plaintext request to a meshed service in it? A. It is rejected with a TLS error B. It is accepted, which is why permissive mode is only a migration step C. The mesh upgrades it to mTLS transparently D. It is accepted only if the pod has a ServiceAccount **Answer: B.** Permissive servers accept both mTLS and plaintext. Meshed clients automatically use mTLS, but nothing stops an unmeshed (or attacker-controlled) pod from connecting in plaintext until you switch to STRICT. 2. What does the SPIFFE ID `spiffe://cluster.local/ns/orders/sa/orders-api` identify? A. The pod's IP address within the cluster B. The node the pod is running on C. The workload running as the `orders-api` ServiceAccount in the `orders` namespace D. The Service object named orders-api **Answer: C.** Istio derives identity from namespace and ServiceAccount, and embeds it in the workload's short-lived certificate. Every pod running as that ServiceAccount shares the identity, which is why each workload needs its own ServiceAccount. 3. You add an Istio AuthorizationPolicy with action ALLOW to the `payments-api` pods, allowing only POST from `orders-api`. A Prometheus pod then fails to scrape `/metrics`. Why? A. Once an ALLOW policy selects a workload, requests matching none of its rules are denied B. Prometheus cannot use mTLS C. AuthorizationPolicy only applies to GET requests D. The policy must be in istio-system to take effect **Answer: A.** ALLOW policies are allow-lists. With one in place, anything not explicitly allowed, including Prometheus's GET /metrics, is rejected. Add a rule for Prometheus's principal. 4. A team runs four services, owned by one team, and wants internal traffic encrypted between nodes. Which is the most proportionate option? A. Install Istio in sidecar mode with STRICT mTLS and full L7 policies B. Move all four services into one pod C. Rely on Kubernetes Secrets to encrypt traffic D. Enable the CNI's node-to-node WireGuard encryption, plus NetworkPolicies to restrict who talks to whom **Answer: D.** A mesh's operational cost is hard to justify for a few services and one team when encryption is the main goal. CNI-level encryption covers that cheaply, and NetworkPolicy handles reachability. A mesh makes sense as identity-based L7 policy and scale become requirements.