Containers & container security

Containers are how almost all modern software is packaged and shipped, and they are the unit everything else in this stage builds on: Kubernetes schedules them, load balancers route to them, autoscalers add more of them. This lesson starts with what a container actually is, then secures both halves of it. An image is a little operating system you ship with every release, attackable through outdated packages and leftover tools; a running container is a process sharing the host's kernel, where excess privileges turn a bug in your app into control of the host. You'll shrink and harden an image with multi-stage builds, distroless bases and a non-root user, scan it with Trivy, and run it with a read-only filesystem, no capabilities and resource limits.

#What a container actually is

A container is not a small virtual machine. It's an ordinary Linux process, started with a few kernel features that change what it can see and use. Run ps on the host and your containerised web server is right there in the process list.

Two kernel features do the work:

• Namespaces give the process its own view of the system. In its own PID namespace it sees itself as PID 1 and no other processes; in its own mount namespace it sees its own root filesystem; network, hostname (UTS), IPC and user namespaces isolate the rest.
• Control groups (cgroups) limit what it can consume: CPU time, memory, number of processes, disk I/O. When a container exceeds its memory limit, the kernel's OOM killer ends it.

Inside its PID namespace, a container sees only itself:

$ docker run --rm alpine ps
PID   USER     TIME  COMMAND
    1 root      0:00 ps

The filesystem it sees comes from an image: a stack of read-only layers, each recording the files added, changed or deleted by one build step, plus metadata such as the default command, user and environment. Layers are content-addressed by their SHA-256 digest, so images that share a base share those layers on disk and in transit. When a container starts, the runtime stacks the layers with an overlay filesystem and adds one thin writable layer on top. Anything the container writes goes there and disappears when the container is removed.

So an image is the immutable template and a container is a running (or stopped) instance of it, the way a program relates to a process. One image can back many containers at once.

Images are stored in registries such as Docker Hub, GitHub Container Registry or your cloud's registry, and named registry/repository:tag, for example ghcr.io/acme/orders-api:1.4.2. A tag is a movable label that anyone with push rights can point at a different image; a digest (@sha256:4c1e...) always identifies exactly the same bytes. The image format and registry API are open OCI standards, which is why an image built with Docker runs unchanged under containerd, Podman or Kubernetes.

#The container threat model

Containers are isolated by Linux kernel features, not by a hypervisor. Namespaces control what the process can see and cgroups what it can consume, as you've just seen; capabilities and seccomp restrict which privileged kernel operations it may perform. If any of those is loosened, isolation weakens. Because every container on a host shares one kernel, a kernel exploit or a misconfiguration can affect all of them.

Most real-world container compromises are less exotic than kernel exploits. They follow a pattern:

1. An attacker exploits your application, or a vulnerable library in the image, to run commands.
2. They use what the image provides (a shell, curl, a package manager) to download tools.
3. They use what the container was granted (root, capabilities, a mounted Docker socket, a writable filesystem) to persist or escape.

Every hardening step in this lesson removes something from steps 2 and 3.

#Choosing a minimal base image

Every package in an image is code an attacker can use and a CVE you have to patch. Your application usually needs a language runtime, CA certificates and time zone data, and nothing else.

Distroless images remove the shell, apt and coreutils. An attacker who gets code execution has no sh to run commands in and nothing to install tools with. Vulnerability scanners also report far fewer findings, so the ones that remain get attention. Hardened image catalogues such as Chainguard's Wolfi-based images and Docker Hardened Images follow the same idea.

The trade-off is debugging: you can't docker exec -it … sh. Use the :debug variant of distroless locally, or attach an ephemeral debug container in Kubernetes, instead of shipping a shell to production.

#Multi-stage builds

A multi-stage build compiles and installs in a full-featured image, then copies only the results into the minimal one. Compilers, dev dependencies, caches and source files never reach the final image.

# syntax=docker/dockerfile:1

# ---- deps: production dependencies only ----
FROM node:24-trixie-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci --omit=dev --ignore-scripts

# ---- build: compile TypeScript with dev dependencies ----
FROM node:24-trixie-slim AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci
COPY tsconfig.json ./
COPY src ./src
RUN npm run build

# ---- runtime: distroless, non-root ----
FROM gcr.io/distroless/nodejs24-debian13:nonroot
WORKDIR /app
ENV NODE_ENV=production
COPY --from=deps  /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
USER 65532:65532
EXPOSE 8080
CMD ["dist/server.js"]

Details that matter:

• Secrets via BuildKit mounts. The private registry token in .npmrc is mounted only for the RUN step (docker build --secret id=npmrc,src=$HOME/.npmrc .). An ARG or COPY would bake it into a layer, where docker history or anyone pulling the image can read it.
• Two dependency stages. The runtime gets node_modules built with --omit=dev, so test frameworks and compilers aren't shipped.
• CMD is just the script. The distroless Node image's entrypoint is already the node binary, and the exec form means Node is PID 1 and receives SIGTERM directly.
• A .dockerignore with .git, .env, node_modules and *.pem keeps local junk and secrets out of the build context entirely.

#Running as a non-root user

By default a container's process runs as root (UID 0). Namespaces mean that's not quite the host's root, but it is the same UID: if the container escapes, or a host directory is mounted in, the process acts as root on the host. Root inside the container can also overwrite the application's own files.

Set a USER in every image. Use a numeric UID and GID: Kubernetes' runAsNonRoot check can only verify a number, not a name. Distroless :nonroot images already include UID 65532. On Debian- or Alpine-based images, create one:

FROM python:3.13-slim
RUN groupadd --system --gid 10001 app \
 && useradd --system --uid 10001 --gid app --no-create-home app
WORKDIR /app
COPY --chown=root:root . .
USER 10001:10001
CMD ["python", "-m", "orders"]

Note --chown=root:root: the application code is owned by root and readable by the app user, but not writable by it. A compromised process can't modify its own code. Non-root processes traditionally can't bind ports below 1024 (Docker and containerd 2.x lift this inside the container's network namespace, but not every runtime does), so listen on 8080 and map it externally.

#Scanning images with Trivy

Trivy scans images for vulnerable OS packages and language dependencies, plus embedded secrets and Dockerfile misconfigurations. Scan the built image, not just the repository, because the base image contributes packages no lockfile lists.

docker build -t orders-api:1.4.2 .
trivy image --severity HIGH,CRITICAL --ignore-unfixed orders-api:1.4.2

orders-api:1.4.2 (debian 13.1)
==============================
Total: 0 (HIGH: 0, CRITICAL: 0)

Node.js (node-pkg)
==================
Total: 3 (HIGH: 3, CRITICAL: 0)

┌─────────────────────────────┬────────────────┬──────────┬────────┬───────────────────┬───────────────┬──────────────────────────────────────────────┐
│           Library           │ Vulnerability  │ Severity │ Status │ Installed Version │ Fixed Version │                    Title                     │
├─────────────────────────────┼────────────────┼──────────┼────────┼───────────────────┼───────────────┼──────────────────────────────────────────────┤
│ braces (package.json)       │ CVE-2024-4068  │ HIGH     │ fixed  │ 3.0.2             │ 3.0.3         │ braces: fails to limit the number of         │
│                             │                │          │        │                   │               │ characters it can handle                     │
├─────────────────────────────┼────────────────┼──────────┼────────┼───────────────────┼───────────────┼──────────────────────────────────────────────┤
│ cross-spawn (package.json)  │ CVE-2024-21538 │ HIGH     │ fixed  │ 7.0.3             │ 7.0.5         │ cross-spawn: regular expression denial of    │
│                             │                │          │        │                   │               │ service                                      │
├─────────────────────────────┼────────────────┼──────────┼────────┼───────────────────┼───────────────┼──────────────────────────────────────────────┤
│ path-to-regexp              │ CVE-2024-45296 │ HIGH     │ fixed  │ 0.1.7             │ 0.1.10        │ path-to-regexp: Backtracking regular         │
│ (package.json)              │                │          │        │                   │               │ expressions cause ReDoS                      │
└─────────────────────────────┴────────────────┴──────────┴────────┴───────────────────┴───────────────┴──────────────────────────────────────────────┘

Reading this output: the distroless OS layer is clean, and all three findings are in npm packages with a fixed version available, so the remedy is npm update (or a lockfile override for a transitive package) and a rebuild. --ignore-unfixed hides vulnerabilities with no patch yet, which you can't act on by upgrading; track them separately rather than failing builds on them.

Useful variations:

# Fail the CI job (exit code 1) if anything HIGH or CRITICAL is fixable
trivy image --exit-code 1 --severity HIGH,CRITICAL --ignore-unfixed orders-api:1.4.2

# Also look for secrets accidentally baked into layers
trivy image --scanners vuln,secret orders-api:1.4.2

# Lint the Dockerfile itself (root user, missing USER, ADD of remote URLs...)
trivy config Dockerfile

Accepted risks go in a .trivyignore file in the repository, one ID per line, with a comment saying why and until when, so exceptions are reviewed in pull requests and revisited. In CI, run the scan in the same job that builds the image, before it's pushed, and again on a schedule against images already in your registry, because new CVEs are published against old images every day. On GitLab the scan is the same command in a job that runs after the build.

#Runtime hardening

A good image can still be run badly. The runtime settings below apply equally to docker run, Docker Compose and Kubernetes securityContext (covered in the next lesson).

Drop capabilities. Linux splits root's power into about 40 capabilities. Docker grants a default set of 14 (including CHOWN, SETUID, NET_RAW and DAC_OVERRIDE). You can see them in a process's effective capability mask:

docker run --rm alpine grep CapEff /proc/self/status
docker run --rm --cap-drop ALL alpine grep CapEff /proc/self/status

CapEff:	00000000a80425fb
CapEff:	0000000000000000

Most web services need none. Drop all, and add back only what's proven necessary, such as NET_BIND_SERVICE if you really must listen on port 80.

Read-only root filesystem. If the app can't write to its own filesystem, an attacker can't drop a binary or modify code. Give it a small in-memory tmpfs for scratch space:

docker run --rm --read-only alpine touch /evil

touch: /evil: Read-only file system

No privilege escalation. --security-opt no-new-privileges stops setuid binaries (like sudo or su) from granting more privileges than the process started with.

Resource limits. Without them, one compromised or buggy container can exhaust the host's memory, CPU or process table (a fork bomb). Set memory, CPU and PID limits.

Putting it together:

docker run -d --name orders \
  --user 65532:65532 \
  --read-only --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  --memory 512m --cpus 1 --pids-limit 200 \
  -p 8080:8080 \
  orders-api:1.4.2

The same in Compose:

services:
  orders:
    image: orders-api:1.4.2
    user: "65532:65532"
    read_only: true
    tmpfs:
      - /tmp:rw,noexec,nosuid,size=64m
    cap_drop: [ALL]
    security_opt: ["no-new-privileges:true"]
    mem_limit: 512m
    cpus: 1
    pids_limit: 200
    ports: ["8080:8080"]

Docker's default seccomp profile, which blocks around 44 rarely needed and dangerous system calls, is on unless someone passes --security-opt seccomp=unconfined; never disable it to "fix" an error without understanding the call being blocked. For defence in depth, rootless Docker or Podman run the daemon itself without root, so even a full escape lands as an unprivileged user.

#Practice

Here is a Dockerfile and run command for a small Python API. List every problem, then rewrite both so the image is minimal, non-root and free of secrets, and the container runs with a read-only filesystem, no capabilities and limits. Finally, scan your image with Trivy so it fails on fixable HIGH or CRITICAL vulnerabilities.

FROM python:3.13
ARG PYPI_TOKEN
COPY . /app
WORKDIR /app
RUN pip install --index-url https://__token__:${PYPI_TOKEN}@pypi.acme.internal/simple -r requirements.txt
CMD python -m orders

docker run -d -p 8080:8080 --privileged orders-api

# syntax=docker/dockerfile:1
FROM python:3.13-slim AS build
WORKDIR /app
RUN python -m venv /venv
COPY requirements.txt .
RUN --mount=type=secret,id=pip_index_url \
    /venv/bin/pip install --no-cache-dir \
      --index-url "$(cat /run/secrets/pip_index_url)" -r requirements.txt
COPY orders ./orders

FROM python:3.13-slim
RUN groupadd --system --gid 10001 app \
 && useradd --system --uid 10001 --gid app --no-create-home app
COPY --from=build /venv /venv
COPY --from=build --chown=root:root /app/orders /app/orders
WORKDIR /app
ENV PATH="/venv/bin:$PATH" PYTHONDONTWRITEBYTECODE=1
USER 10001:10001
EXPOSE 8080
CMD ["python", "-m", "orders"]

docker build --secret id=pip_index_url,env=PIP_INDEX_URL -t orders-api:1.0.0 .
trivy image --exit-code 1 --severity HIGH,CRITICAL --ignore-unfixed orders-api:1.0.0

docker run -d --name orders -p 8080:8080 \
  --read-only --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  --cap-drop ALL --security-opt no-new-privileges \
  --memory 256m --cpus 0.5 --pids-limit 100 \
  orders-api:1.0.0

#Key takeaways

• Containers share the host kernel; every privilege and tool you leave in is something an attacker can use after exploiting your app.
• Use multi-stage builds so compilers, dev dependencies and source never reach the final image, and pass build secrets with BuildKit secret mounts, never ARG.
• Prefer distroless or similarly minimal bases, and run as a numeric non-root UID with code owned by root.
• Scan the built image with Trivy, fail on fixable HIGH/CRITICAL findings, record exceptions in .trivyignore, and rebuild regularly.
• Run with --cap-drop ALL, a read-only root filesystem plus tmpfs, no-new-privileges and memory, CPU and PID limits.
• Treat --privileged, a mounted Docker socket and host namespaces as root on the host.