# Kubernetes fundamentals

> Deploy, expose, configure and update a web app on Kubernetes, and inspect and debug it with kubectl.

DevSecOps Engineer roadmap · Run the platform · 24 min read · https://levelcode.org/roadmaps/devsecops/kubernetes-fundamentals

You can build an image and run it with `docker run`. Production needs more than that: several copies spread over several machines, restarted when they crash, replaced one at a time on every release, and reachable at a stable address while all of that happens. **Kubernetes** is the system most teams use to do this. In this lesson you'll learn how a cluster is put together and how it thinks, then deploy a web app with a Deployment, expose it with a Service and an Ingress, configure it with ConfigMaps and Secrets, keep it healthy with resource settings and probes, and inspect all of it with `kubectl`. Security hardening of these objects comes later, in the Kubernetes security lesson.

## Desired state and the cluster

With plain Docker you give **imperative** commands: start this container, stop that one. If a container dies at 3 a.m., nothing brings it back unless you scripted it. Kubernetes is **declarative**: you describe the state you want ("three copies of `orders-api:1.4.2`, reachable on port 80") and store that description in the cluster. Programs called **controllers** then run a **reconciliation loop** forever:

1. **Observe** the actual state (two pods are running).
2. **Compare** it with the desired state (three should be).
3. **Act** to close the gap (create one pod), then go back to step 1.

This one idea explains most of Kubernetes' behaviour. Delete a pod by hand and a replacement appears seconds later, because the desired state still says three. Lose a whole machine and its pods are recreated elsewhere. And "deploying" is just changing the desired state; the controllers work out the steps.

### The control plane

The **control plane** stores the desired state and runs the controllers:

- **kube-apiserver** is the front door. Every client (you with `kubectl`, the controllers, the nodes) talks only to it, over HTTPS. It authenticates and authorises each request, runs admission checks, and is the only component that reads and writes etcd.
- **etcd** is a consistent, replicated key-value store holding every object in the cluster. Lose etcd without a backup and you lose the cluster's memory.
- **kube-scheduler** watches for pods that have no node yet and picks one, based on the resources each pod requests, what each node has free, and any placement rules.
- **kube-controller-manager** runs the built-in controllers: the Deployment controller, ReplicaSet controller, node controller and dozens more. In a cloud, a **cloud-controller-manager** also creates load balancers and handles node lifecycle for that provider.

### The nodes

**Nodes** are the machines that run your workloads. Each one runs:

- **kubelet**, the node agent. It watches the API server for pods assigned to its node, tells the container runtime to start them, runs their health probes, and reports their status back.
- A **container runtime**, usually containerd or CRI-O, which pulls images and starts containers through the Container Runtime Interface (CRI). Kubernetes stopped talking to Docker Engine directly in version 1.24, but images you build with Docker are standard OCI images and run unchanged.
- **kube-proxy**, which programs each node's packet rules (iptables or nftables) so that traffic to a Service's virtual IP reaches a healthy pod. Some network plugins, such as Cilium, replace it.

A **CNI plugin** (Calico, Cilium, a cloud's own) gives every pod an IP address and connects pods across nodes. In a managed cluster (EKS, GKE, AKS) the provider runs the control plane and you see only the nodes. In a local **kind** cluster, everything runs in Docker containers on your laptop:

```text
$ kubectl get pods -n kube-system
NAME                                         READY   STATUS    RESTARTS   AGE
coredns-7c65d6cfc9-5zq8w                     1/1     Running   0          4m
coredns-7c65d6cfc9-x2l9d                     1/1     Running   0          4m
etcd-kind-control-plane                      1/1     Running   0          4m
kindnet-8tkbh                                1/1     Running   0          4m
kube-apiserver-kind-control-plane            1/1     Running   0          4m
kube-controller-manager-kind-control-plane   1/1     Running   0          4m
kube-proxy-7xw2r                             1/1     Running   0          4m
kube-scheduler-kind-control-plane            1/1     Running   0          4m
```

### What happens on `kubectl apply`

Follow one Deployment from your terminal to a running container. Notice that no component tells another what to do directly: each one writes to the API server, and the next one is *watching* the API server for exactly that change.

```sequence
title: From kubectl apply to a running container
participants: kubectl, API server, etcd, Scheduler, Kubelet, Runtime
kubectl -> API server: PATCH Deployment orders-api
> kubectl sends the manifest over HTTPS. The API server authenticates you, checks RBAC, runs admission controllers and validates the object against the apps/v1 schema.
API server -> etcd: Store desired state
> Only now is the change real. etcd holds the new desired state: three replicas of orders-api:1.4.2.
API server --> kubectl: deployment.apps/orders-api configured
> kubectl returns immediately. Nothing is running yet; the rest happens asynchronously.
note API server: Controllers: Deployment, ReplicaSet, Pods
> The controller manager is watching Deployments. It creates a ReplicaSet, the ReplicaSet controller sees too few pods and creates three Pod objects, each with an empty nodeName.
Scheduler -> API server: Bind pod to worker-2
> The scheduler watches for unscheduled pods, filters out nodes without enough free CPU and memory for the pod's requests, scores the rest, and writes its choice back.
Kubelet -> API server: Watch: pod bound to my node
> The kubelet on worker-2 sees a pod assigned to it that isn't running yet.
Kubelet -> Runtime: Pull image, create and start container
> Through the CRI, containerd pulls the image if needed, sets up namespaces and cgroups, and starts the process. The CNI plugin gives the pod its IP.
Runtime --> Kubelet: Container running
Kubelet -> Kubelet: Run startup and readiness probes
> The pod counts as Ready only when its readiness probe passes.
Kubelet -> API server: Status: Running, Ready
> Ready pods are added to their Service's endpoints. If the container later dies, the kubelet restarts it; if the node dies, the ReplicaSet controller replaces the pod elsewhere. That is reconciliation.
```

## Pods, labels and namespaces

A **Pod** is the smallest thing Kubernetes runs: one or more containers that share a network namespace (one IP, one `localhost`) and can share volumes. Most pods have one application container; extra containers are helpers such as a log shipper or a proxy.

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: hello
  namespace: orders
  labels:
    app: hello
spec:
  containers:
    - name: web
      image: traefik/whoami:v1.12.0
      ports:
        - containerPort: 80
```

Pods are **ephemeral**. When a node fails or a pod is replaced, the new pod has a new name and a new IP. That's why you almost never create pods directly: you create a controller that creates them for you, and a Service that gives them a stable address.

**Labels** are key-value pairs on any object (`app: orders-api`, `tier: backend`, `env: prod`). **Selectors** query them, and they are how objects find each other: a Deployment finds its pods by label, a Service finds its endpoints by label, a NetworkPolicy chooses which pods it applies to by label. A typo in a label is one of the most common reasons "nothing is happening".

```text
$ kubectl get pods -n orders -l app=orders-api --show-labels
NAME                          READY   STATUS    RESTARTS   AGE   LABELS
orders-api-7d9c6b8f4c-2xkqp   1/1     Running   0          8m    app=orders-api,pod-template-hash=7d9c6b8f4c
orders-api-7d9c6b8f4c-9mfzt   1/1     Running   0          8m    app=orders-api,pod-template-hash=7d9c6b8f4c
orders-api-7d9c6b8f4c-tq4wn   1/1     Running   0          8m    app=orders-api,pod-template-hash=7d9c6b8f4c
```

**Namespaces** partition a cluster into named groups (`orders`, `payments`, `monitoring`). Names only need to be unique within a namespace, and namespaces are the unit for RBAC permissions, resource quotas, Pod Security settings and network policies. A few objects, such as Nodes and Namespaces themselves, are cluster-scoped. Use `-n <namespace>` on every command, or set a default with `kubectl config set-context --current --namespace=orders`.

## Deployments: rolling updates and rollbacks

A **Deployment** is how you run a stateless app. You give it a pod template and a replica count; it manages a **ReplicaSet**, which keeps exactly that many pods running.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  namespace: orders
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api            # must match the template's labels
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1                # at most 1 pod above 3 during an update
      maxUnavailable: 0          # never fewer than 3 ready pods
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      containers:
        - name: api
          image: ghcr.io/acme/orders-api:1.4.2
          ports:
            - name: http
              containerPort: 8080
```

Why two layers? Each ReplicaSet represents one version of the pod template. When you change the template (a new image, a new environment variable), the Deployment creates a new ReplicaSet and shifts pods across: start one new pod, wait until it's Ready, remove one old pod, repeat. With `maxUnavailable: 0` capacity never drops during a release. The old ReplicaSet is kept, scaled to zero, which is what makes rollbacks instant.

Ship a new version by changing the image in the manifest and applying it, then watch the rollout:

```text
$ kubectl apply -f deployment.yaml
deployment.apps/orders-api configured
$ kubectl rollout status deployment/orders-api -n orders
Waiting for deployment "orders-api" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "orders-api" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "orders-api" rollout to finish: 1 old replicas are pending termination...
deployment "orders-api" successfully rolled out
$ kubectl get replicasets -n orders
NAME                    DESIRED   CURRENT   READY   AGE
orders-api-5f7b9c7d8b   3         3         3       40s
orders-api-7d9c6b8f4c   0         0         0       9m
```

Now a bad release: version `1.4.3` was never pushed. The first new pod can't pull its image, and because it never becomes Ready, the rollout stops there. The three old pods keep serving traffic:

```text
$ kubectl get pods -n orders
NAME                          READY   STATUS             RESTARTS   AGE
orders-api-5f7b9c7d8b-8hx4m   1/1     Running            0          3m
orders-api-5f7b9c7d8b-kq2vw   1/1     Running            0          3m
orders-api-5f7b9c7d8b-zt7np   1/1     Running            0          3m
orders-api-6c4d8f9b7d-r2wjc   0/1     ImagePullBackOff   0          50s
$ kubectl rollout undo deployment/orders-api -n orders
deployment.apps/orders-api rolled back
```

`rollout undo` scales the previous ReplicaSet back up. `kubectl rollout history deployment/orders-api` lists revisions, and `--to-revision=N` picks a specific one. If a rollout makes no progress for `progressDeadlineSeconds` (600 by default), the Deployment is marked failed and `rollout status` exits non-zero, which is how a CI deploy job notices.

> [!WARNING]
> `rollout undo` changes the cluster, not your manifests. If Git still says `1.4.3`, the next `kubectl apply` or GitOps sync breaks production again. Fix the source of truth too. Also, a Deployment's `selector` is immutable after creation: changing it means deleting and recreating the Deployment.

## Services, DNS and Ingress

Pods come and go; clients need one address that always works. A **Service** selects pods by label and gives them a stable virtual IP and DNS name. Kubernetes keeps a list of the Ready pods' IPs in **EndpointSlices**, and kube-proxy load-balances connections across them.

```yaml
apiVersion: v1
kind: Service
metadata:
  name: orders-api
  namespace: orders
spec:
  type: ClusterIP
  selector:
    app: orders-api
  ports:
    - name: http
      port: 80            # the port clients connect to
      targetPort: http    # the container port, by name
```

```text
$ kubectl get service,endpointslices -n orders
NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/orders-api   ClusterIP   10.96.142.17   <none>        80/TCP    5m

NAME                                              ADDRESSTYPE   PORTS   ENDPOINTS                          AGE
endpointslice.discovery.k8s.io/orders-api-8xq2d   IPv4          8080    10.244.1.7,10.244.2.4,10.244.2.9   5m
```

If `ENDPOINTS` is empty, the selector matches no Ready pods: check the labels and the readiness probe. The cluster DNS (CoreDNS) gives every Service a name of the form `<service>.<namespace>.svc.cluster.local`. Pods in the same namespace can just use `http://orders-api`; from another namespace, use `orders-api.orders`.

The `type` field controls how far the Service is reachable:

| Type | Reachable from | How |
|---|---|---|
| `ClusterIP` (default) | Inside the cluster only | A virtual IP and DNS name |
| `NodePort` | Outside, via any node's IP | Also opens a port in 30000–32767 on every node |
| `LoadBalancer` | Outside, via a cloud load balancer | Also asks the cloud to provision an L4 load balancer pointing at the node ports |

`LoadBalancer` gives every Service its own external load balancer and IP, which gets expensive, and it only understands TCP and UDP. On kind or minikube with no cloud, its `EXTERNAL-IP` stays `<pending>` unless you add a helper such as `minikube tunnel` or cloud-provider-kind.

### Ingress

An **Ingress** routes HTTP and HTTPS by hostname and path to Services, behind one entry point, and can terminate TLS:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: orders
  namespace: orders
spec:
  ingressClassName: traefik
  tls:
    - hosts: ["shop.example.com"]
      secretName: shop-tls          # a kubernetes.io/tls Secret
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /api/orders
            pathType: Prefix
            backend:
              service:
                name: orders-api
                port:
                  name: http
```

An Ingress object is only a routing rule. Something has to implement it: an **Ingress controller** such as Traefik, HAProxy Ingress or your cloud's load balancer controller, selected by `ingressClassName`. Without one installed, Ingress objects do nothing at all. The once-dominant ingress-nginx project was retired by the Kubernetes project in March 2026 and no longer receives security fixes, so don't choose it for new clusters. Ingress is stable but frozen; its successor, the **Gateway API** (`Gateway` and `HTTPRoute` objects), adds header matching, traffic splitting and a cleaner split between platform and app teams. It's covered in the API gateways lesson.

## ConfigMaps and Secrets

The same image should run in every environment, so configuration lives outside it. A **ConfigMap** holds non-sensitive settings; a **Secret** holds credentials. Pods consume both as environment variables or as files:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: orders-config
  namespace: orders
data:
  LOG_LEVEL: info
  PAYMENTS_URL: http://payments-api.payments
---
apiVersion: v1
kind: Secret
metadata:
  name: orders-db
  namespace: orders
type: Opaque
stringData:                      # plain text in; stored base64-encoded
  password: dev-only-password
```

In the Deployment's container spec:

```yaml
          envFrom:
            - configMapRef:
                name: orders-config      # every key becomes a variable
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: orders-db
                  key: password
```

Two behaviours surprise people. First, environment variables are read when the container starts, so editing a ConfigMap changes nothing in running pods; run `kubectl rollout restart deployment/orders-api` to roll new pods. Files mounted from a ConfigMap or Secret volume are updated in place after a short delay, which suits apps that reload their config. Second, a Secret is **not encrypted** by Kubernetes: `stringData` and `data` are only base64-encoded, so a manifest like the one above must never be committed. In real clusters the values come from a secret manager, through the patterns in the secrets management lesson's "Secrets in Kubernetes" section; who can read them, and how they're protected in etcd, is covered in the Kubernetes security lesson.

## Resources and health probes

Kubernetes can only place and protect your pods well if you tell it what they need and how to check them.

**Requests and limits.** A **request** is what the scheduler reserves for the container: a pod requesting `cpu: 250m` (a quarter of a core) and `memory: 256Mi` only lands on a node with that much unreserved. A **limit** is a ceiling enforced by cgroups at runtime: a container over its CPU limit is throttled (slowed), but one over its memory limit is killed.

```yaml
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              memory: 512Mi
```

```text
$ kubectl get pods -n orders
NAME                          READY   STATUS      RESTARTS      AGE
orders-api-5f7b9c7d8b-8hx4m   0/1     OOMKilled   3 (41s ago)   6m
```

`OOMKilled` (exit code 137) means the memory limit is too low or the app leaks. Pods with no requests at all get the `BestEffort` quality-of-service class: the scheduler packs them anywhere, and they are the first evicted when a node runs short. Always set requests; set a memory limit; CPU limits are optional and many teams leave them off to avoid throttling.

**Probes.** The kubelet checks each container with up to three probes, and they do different things:

| Probe | Question | On failure |
|---|---|---|
| `startupProbe` | Has the app finished starting? | Keeps checking until `failureThreshold`; the other probes wait. Then restarts the container |
| `readinessProbe` | Should this pod receive traffic now? | Removes the pod from Service endpoints; no restart |
| `livenessProbe` | Is the process stuck beyond recovery? | Restarts the container |

```yaml
          startupProbe:
            httpGet: { path: /healthz, port: http }
            periodSeconds: 5
            failureThreshold: 24       # up to 2 minutes to start
          readinessProbe:
            httpGet: { path: /ready, port: http }
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet: { path: /healthz, port: http }
            periodSeconds: 10
            failureThreshold: 3
```

Readiness is what makes rolling updates safe: a new pod receives no traffic, and the rollout doesn't continue, until it reports ready. The classic mistake is a liveness probe that checks dependencies. If `/healthz` queries the database and the database has a hiccup, every pod fails liveness and restarts at once, turning a brief blip into an outage. Liveness should answer only "is this process alive?"; put dependency checks, if any, in readiness.

## kubectl essentials

`kubectl` reads your cluster credentials from `~/.kube/config`; `kubectl config current-context` tells you which cluster you're about to change, so check it before anything destructive. The commands below cover most daily work.

**`apply`** sends manifests and creates or updates objects. `kubectl diff -f` shows what would change first.

```text
$ kubectl apply -f k8s/
configmap/orders-config created
deployment.apps/orders-api created
service/orders-api created
ingress.networking.k8s.io/orders created
```

**`get`** lists objects; `-o wide` adds pod IPs and nodes, `-o yaml` shows the full stored object, `-w` watches for changes.

```text
$ kubectl get pods -n orders -o wide
NAME                          READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
orders-api-7d9c6b8f4c-2xkqp   1/1     Running   0          8m    10.244.1.7   worker-1   <none>           <none>
orders-api-7d9c6b8f4c-9mfzt   1/1     Running   0          8m    10.244.2.4   worker-2   <none>           <none>
orders-api-7d9c6b8f4c-tq4wn   0/1     Running   0          8m    10.244.2.9   worker-2   <none>           <none>
```

**`describe`** is the first stop when something's wrong, because it ends with the object's **events**. The third pod above is running but not ready:

```text
$ kubectl describe pod orders-api-7d9c6b8f4c-tq4wn -n orders
...
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  8m                  default-scheduler  Successfully assigned orders/orders-api-7d9c6b8f4c-tq4wn to worker-2
  Normal   Started    8m                  kubelet            Started container api
  Warning  Unhealthy  12s (x94 over 8m)   kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503
```

**`logs`** prints a container's output. `-f` follows, `--previous` shows the crashed instance before a restart (essential for `CrashLoopBackOff`), and `deployment/orders-api` picks one of its pods for you.

```text
$ kubectl logs orders-api-7d9c6b8f4c-tq4wn -n orders --tail=3
{"level":"info","msg":"listening","port":8080}
{"level":"error","msg":"readiness: payments-api unreachable","err":"dial tcp 10.96.201.33:80: i/o timeout"}
{"level":"error","msg":"readiness: payments-api unreachable","err":"dial tcp 10.96.201.33:80: i/o timeout"}
```

**`rollout`** manages Deployments: `status`, `history`, `undo` and `restart`, as you saw above.

**`port-forward`** tunnels a local port to a pod or Service through the API server, for testing something that isn't exposed:

```text
$ kubectl port-forward -n orders svc/orders-api 8080:80
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
```

A useful habit when debugging: work outside-in. Is the Ingress pointing at the right Service? Does the Service have endpoints? Are the pods Ready, and if not, what do `describe` and `logs` say?

## Practice

On a local cluster, deploy `traefik/whoami:v1.12.0`, a tiny web server that prints which pod answered and has a `/health` endpoint whose status code you can change with a `POST`. Then:

1. Create a kind cluster (`kind create cluster`) or start minikube, and a namespace `hello`.
2. Write a ConfigMap setting `WHOAMI_NAME=hello-v1` and `WHOAMI_PORT_NUMBER=8080`, and a Deployment with 3 replicas that loads it, with requests, a memory limit, and readiness and liveness probes on `/health`.
3. Add a ClusterIP Service on port 80 and an Ingress for host `hello.local`. Install Traefik as the Ingress controller with Helm (`helm install traefik traefik/traefik -n traefik --create-namespace` after `helm repo add traefik https://traefik.github.io/charts`) and reach the app through it.
4. Confirm requests are spread across pods. Make one pod's `/health` return 500 and watch what readiness and liveness do.
5. Roll out a broken image tag, observe the stalled rollout, and roll back.

**Solution**

```yaml
# hello.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: hello
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: hello-config
  namespace: hello
data:
  WHOAMI_NAME: hello-v1
  WHOAMI_PORT_NUMBER: "8080"       # ConfigMap values are always strings
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  namespace: hello
spec:
  replicas: 3
  selector:
    matchLabels: { app: hello }
  strategy:
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }
  template:
    metadata:
      labels: { app: hello }
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        seccompProfile: { type: RuntimeDefault }
      containers:
        - name: web
          image: traefik/whoami:v1.12.0
          envFrom:
            - configMapRef: { name: hello-config }
          ports:
            - name: http
              containerPort: 8080
          resources:
            requests: { cpu: 50m, memory: 32Mi }
            limits: { memory: 64Mi }
          readinessProbe:
            httpGet: { path: /health, port: http }
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet: { path: /health, port: http }
            periodSeconds: 10
            failureThreshold: 3
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
---
apiVersion: v1
kind: Service
metadata:
  name: hello
  namespace: hello
spec:
  selector: { app: hello }
  ports:
    - name: http
      port: 80
      targetPort: http
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello
  namespace: hello
spec:
  ingressClassName: traefik
  rules:
    - host: hello.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: hello
                port: { name: http }
```

```bash
kind create cluster
helm repo add traefik https://traefik.github.io/charts && helm repo update
helm install traefik traefik/traefik -n traefik --create-namespace
kubectl apply -f hello.yaml
kubectl rollout status deployment/hello -n hello
kubectl get svc -n traefik     # LoadBalancer, EXTERNAL-IP <pending> on kind: expected

# Terminal 2: reach Traefik's Service from your machine
kubectl port-forward -n traefik svc/traefik 8080:80

# Terminal 1: the Host header selects the Ingress rule
for i in 1 2 3 4 5 6; do curl -s -H "Host: hello.local" http://localhost:8080/ | grep Hostname; done
```

```text
Hostname: hello-6d5f7c9b8d-4kx2m
Hostname: hello-6d5f7c9b8d-qz8wn
Hostname: hello-6d5f7c9b8d-4kx2m
Hostname: hello-6d5f7c9b8d-vt5hp
Hostname: hello-6d5f7c9b8d-qz8wn
Hostname: hello-6d5f7c9b8d-vt5hp
```

Break one pod's health (use a real pod name from `kubectl get pods -n hello`):

```bash
kubectl port-forward -n hello pod/hello-6d5f7c9b8d-4kx2m 9000:8080 &
curl -X POST -d '500' http://localhost:9000/health
kubectl get pods -n hello -w
```

Within about 10 seconds that pod shows `0/1` READY and stops appearing in the curl loop: readiness removed it from the Service's endpoints. About 30 seconds after the first failure, liveness restarts the container (`RESTARTS` becomes 1, `describe` shows "Container web failed liveness probe, will be restarted"); the fresh process returns 200 again and the pod rejoins. The port-forward to that pod drops when it restarts.

Now a broken release, and the rollback:

```bash
kubectl set image deployment/hello -n hello web=traefik/whoami:v9.9.9
kubectl rollout status deployment/hello -n hello --timeout=60s   # times out
kubectl get pods -n hello        # one ImagePullBackOff, three old pods Running
kubectl rollout undo deployment/hello -n hello
kubectl rollout status deployment/hello -n hello
```

Because `maxUnavailable: 0`, the curl loop keeps answering throughout. Remember to put the good tag back in `hello.yaml` too. Clean up with `kind delete cluster`.

## Key takeaways

- Kubernetes is declarative: you store desired state in the API server, and controllers continuously reconcile actual state towards it.
- The control plane (API server, etcd, scheduler, controller manager) decides; each node's kubelet and container runtime run the pods; components coordinate only by watching the API server.
- Run stateless apps as Deployments: they manage ReplicaSets, roll out changes gradually behind readiness checks, and roll back with `kubectl rollout undo`.
- Services give label-selected pods a stable IP and DNS name; Ingress (or its successor, the Gateway API) routes external HTTP to Services, and needs a controller to do anything.
- Keep configuration in ConfigMaps and credentials in Secrets, remembering that Secrets are only base64-encoded.
- Set resource requests and a memory limit, use readiness to gate traffic and liveness only for a stuck process, and debug with `get`, `describe`, `logs` and `port-forward`.

## Quiz

1. You delete one of a Deployment's three pods with `kubectl delete pod`. What happens?

   A. The Deployment now runs two pods until you scale it again
   B. The ReplicaSet controller notices only two pods exist and creates a replacement
   C. The kubelet restarts the same pod with the same name and IP
   D. The Deployment is marked failed and rolls back

   **Answer: B.** Desired state still says three replicas. The ReplicaSet controller's reconciliation loop sees the gap and creates a new pod, with a new name and IP, which the scheduler places on a node.

2. A Service exists, but `kubectl get endpointslices` shows no endpoints and requests to it fail. The pods are Running. What is the most likely cause?

   A. The Service type should be LoadBalancer
   B. CoreDNS is not running
   C. The Service's selector doesn't match the pods' labels, or the pods aren't passing their readiness probe
   D. The Ingress controller is missing

   **Answer: C.** Endpoints are the Ready pods whose labels match the selector. An empty list means either the labels don't match or no pod is Ready. DNS and Ingress problems wouldn't empty the endpoint list.

3. A liveness probe calls `/healthz`, which also checks the database connection. The database restarts for 40 seconds. What happens?

   A. Every pod fails liveness and is restarted, turning a short database blip into an application outage
   B. Nothing, because liveness probes only check that the port is open
   C. Pods are removed from the Service but not restarted
   D. The Deployment rolls back to the previous version

   **Answer: A.** Liveness failures restart containers. A dependency check in liveness makes every replica restart together. Liveness should test only the process itself; dependency checks belong in readiness, if anywhere.

4. During a rolling update with `maxUnavailable: 0`, the first new pod is stuck in ImagePullBackOff. What is the state of the application?

   A. All traffic fails because the Deployment is mid-update
   B. Half the pods have been replaced with broken ones
   C. Kubernetes automatically rolls back to the previous version
   D. The old pods keep serving; the rollout waits because the new pod never becomes Ready, until you fix or undo it

   **Answer: D.** Old pods are only removed once new ones are Ready, so capacity is preserved. Kubernetes doesn't roll back on its own: after progressDeadlineSeconds the rollout is marked failed, and you run rollout undo or fix the image.
