# Containers & container security > Build minimal multi-stage images that run as non-root, scan them with Trivy, and lock containers down at runtime with capabilities and limits. DevSecOps Engineer roadmap · Run the platform · 20 min read · https://levelcode.org/roadmaps/devsecops/container-security Containers are how almost all modern software is packaged and shipped, and they are the unit everything else in this stage builds on: Kubernetes schedules them, load balancers route to them, autoscalers add more of them. This lesson starts with what a container actually is, then secures both halves of it. An image is a little operating system you ship with every release, attackable through outdated packages and leftover tools; a running container is a process sharing the host's kernel, where excess privileges turn a bug in your app into control of the host. You'll shrink and harden an image with multi-stage builds, distroless bases and a non-root user, scan it with Trivy, and run it with a read-only filesystem, no capabilities and resource limits. ## What a container actually is A container is not a small virtual machine. It's an ordinary Linux process, started with a few kernel features that change what it can see and use. Run `ps` on the host and your containerised web server is right there in the process list. Two kernel features do the work: - **Namespaces** give the process its own *view* of the system. In its own PID namespace it sees itself as PID 1 and no other processes; in its own mount namespace it sees its own root filesystem; network, hostname (UTS), IPC and user namespaces isolate the rest. - **Control groups (cgroups)** limit what it can *consume*: CPU time, memory, number of processes, disk I/O. When a container exceeds its memory limit, the kernel's OOM killer ends it. Inside its PID namespace, a container sees only itself: ```text $ docker run --rm alpine ps PID USER TIME COMMAND 1 root 0:00 ps ``` The filesystem it sees comes from an **image**: a stack of read-only **layers**, each recording the files added, changed or deleted by one build step, plus metadata such as the default command, user and environment. Layers are content-addressed by their SHA-256 digest, so images that share a base share those layers on disk and in transit. When a container starts, the runtime stacks the layers with an overlay filesystem and adds one thin writable layer on top. Anything the container writes goes there and disappears when the container is removed. So an **image** is the immutable template and a **container** is a running (or stopped) instance of it, the way a program relates to a process. One image can back many containers at once. Images are stored in **registries** such as Docker Hub, GitHub Container Registry or your cloud's registry, and named `registry/repository:tag`, for example `ghcr.io/acme/orders-api:1.4.2`. A tag is a movable label that anyone with push rights can point at a different image; a digest (`@sha256:4c1e...`) always identifies exactly the same bytes. The image format and registry API are open OCI standards, which is why an image built with Docker runs unchanged under containerd, Podman or Kubernetes. ## The container threat model Containers are isolated by Linux kernel features, not by a hypervisor. Namespaces control what the process can see and cgroups what it can consume, as you've just seen; **capabilities** and **seccomp** restrict which privileged kernel operations it may perform. If any of those is loosened, isolation weakens. Because every container on a host shares one kernel, a kernel exploit or a misconfiguration can affect all of them. Most real-world container compromises are less exotic than kernel exploits. They follow a pattern: 1. An attacker exploits your application, or a vulnerable library in the image, to run commands. 2. They use what the image provides (a shell, `curl`, a package manager) to download tools. 3. They use what the container was granted (root, capabilities, a mounted Docker socket, a writable filesystem) to persist or escape. Every hardening step in this lesson removes something from steps 2 and 3. ## Choosing a minimal base image Every package in an image is code an attacker can use and a CVE you have to patch. Your application usually needs a language runtime, CA certificates and time zone data, and nothing else. | Base | Shell and package manager | Relative size | Notes | |---|---|---|---| | `node:24` (Debian) | Yes | Largest | Compilers and build tools included; fine for a *build* stage | | `node:24-trixie-slim` | Yes | Medium | Fewer packages, still a full userland | | `node:24-alpine` | Yes (`apk`) | Small | musl libc; occasionally incompatible with native modules | | `gcr.io/distroless/nodejs24-debian13` | No | Small (mostly the Node binary) | Just Node, glibc, CA certs, tzdata; `:nonroot` tag runs as UID 65532 | | `scratch` | No | Empty | Nothing at all; only for static binaries such as Go with `CGO_ENABLED=0` | **Distroless** images remove the shell, `apt` and coreutils. An attacker who gets code execution has no `sh` to run commands in and nothing to install tools with. Vulnerability scanners also report far fewer findings, so the ones that remain get attention. Hardened image catalogues such as Chainguard's Wolfi-based images and Docker Hardened Images follow the same idea. The trade-off is debugging: you can't `docker exec -it … sh`. Use the `:debug` variant of distroless locally, or attach an ephemeral debug container in Kubernetes, instead of shipping a shell to production. ## Multi-stage builds A **multi-stage build** compiles and installs in a full-featured image, then copies only the results into the minimal one. Compilers, dev dependencies, caches and source files never reach the final image. ```dockerfile # syntax=docker/dockerfile:1 # ---- deps: production dependencies only ---- FROM node:24-trixie-slim AS deps WORKDIR /app COPY package.json package-lock.json ./ RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \ npm ci --omit=dev --ignore-scripts # ---- build: compile TypeScript with dev dependencies ---- FROM node:24-trixie-slim AS build WORKDIR /app COPY package.json package-lock.json ./ RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci COPY tsconfig.json ./ COPY src ./src RUN npm run build # ---- runtime: distroless, non-root ---- FROM gcr.io/distroless/nodejs24-debian13:nonroot WORKDIR /app ENV NODE_ENV=production COPY --from=deps /app/node_modules ./node_modules COPY --from=build /app/dist ./dist USER 65532:65532 EXPOSE 8080 CMD ["dist/server.js"] ``` Details that matter: - **Secrets via BuildKit mounts.** The private registry token in `.npmrc` is mounted only for the `RUN` step (`docker build --secret id=npmrc,src=$HOME/.npmrc .`). An `ARG` or `COPY` would bake it into a layer, where `docker history` or anyone pulling the image can read it. - **Two dependency stages.** The runtime gets `node_modules` built with `--omit=dev`, so test frameworks and compilers aren't shipped. - **`CMD` is just the script.** The distroless Node image's entrypoint is already the `node` binary, and the exec form means Node is PID 1 and receives `SIGTERM` directly. - **A `.dockerignore`** with `.git`, `.env`, `node_modules` and `*.pem` keeps local junk and secrets out of the build context entirely. ## Running as a non-root user By default a container's process runs as **root (UID 0)**. Namespaces mean that's not *quite* the host's root, but it is the same UID: if the container escapes, or a host directory is mounted in, the process acts as root on the host. Root inside the container can also overwrite the application's own files. Set a `USER` in every image. Use a **numeric** UID and GID: Kubernetes' `runAsNonRoot` check can only verify a number, not a name. Distroless `:nonroot` images already include UID 65532. On Debian- or Alpine-based images, create one: ```dockerfile FROM python:3.13-slim RUN groupadd --system --gid 10001 app \ && useradd --system --uid 10001 --gid app --no-create-home app WORKDIR /app COPY --chown=root:root . . USER 10001:10001 CMD ["python", "-m", "orders"] ``` Note `--chown=root:root`: the application code is owned by root and *readable* by the app user, but not writable by it. A compromised process can't modify its own code. Non-root processes traditionally can't bind ports below 1024 (Docker and containerd 2.x lift this inside the container's network namespace, but not every runtime does), so listen on 8080 and map it externally. ## Scanning images with Trivy **Trivy** scans images for vulnerable OS packages and language dependencies, plus embedded secrets and Dockerfile misconfigurations. Scan the built image, not just the repository, because the base image contributes packages no lockfile lists. ```bash docker build -t orders-api:1.4.2 . trivy image --severity HIGH,CRITICAL --ignore-unfixed orders-api:1.4.2 ``` ```text orders-api:1.4.2 (debian 13.1) ============================== Total: 0 (HIGH: 0, CRITICAL: 0) Node.js (node-pkg) ================== Total: 3 (HIGH: 3, CRITICAL: 0) ┌─────────────────────────────┬────────────────┬──────────┬────────┬───────────────────┬───────────────┬──────────────────────────────────────────────┐ │ Library │ Vulnerability │ Severity │ Status │ Installed Version │ Fixed Version │ Title │ ├─────────────────────────────┼────────────────┼──────────┼────────┼───────────────────┼───────────────┼──────────────────────────────────────────────┤ │ braces (package.json) │ CVE-2024-4068 │ HIGH │ fixed │ 3.0.2 │ 3.0.3 │ braces: fails to limit the number of │ │ │ │ │ │ │ │ characters it can handle │ ├─────────────────────────────┼────────────────┼──────────┼────────┼───────────────────┼───────────────┼──────────────────────────────────────────────┤ │ cross-spawn (package.json) │ CVE-2024-21538 │ HIGH │ fixed │ 7.0.3 │ 7.0.5 │ cross-spawn: regular expression denial of │ │ │ │ │ │ │ │ service │ ├─────────────────────────────┼────────────────┼──────────┼────────┼───────────────────┼───────────────┼──────────────────────────────────────────────┤ │ path-to-regexp │ CVE-2024-45296 │ HIGH │ fixed │ 0.1.7 │ 0.1.10 │ path-to-regexp: Backtracking regular │ │ (package.json) │ │ │ │ │ │ expressions cause ReDoS │ └─────────────────────────────┴────────────────┴──────────┴────────┴───────────────────┴───────────────┴──────────────────────────────────────────────┘ ``` Reading this output: the distroless OS layer is clean, and all three findings are in npm packages with a **fixed version** available, so the remedy is `npm update` (or a lockfile override for a transitive package) and a rebuild. `--ignore-unfixed` hides vulnerabilities with no patch yet, which you can't act on by upgrading; track them separately rather than failing builds on them. Useful variations: ```bash # Fail the CI job (exit code 1) if anything HIGH or CRITICAL is fixable trivy image --exit-code 1 --severity HIGH,CRITICAL --ignore-unfixed orders-api:1.4.2 # Also look for secrets accidentally baked into layers trivy image --scanners vuln,secret orders-api:1.4.2 # Lint the Dockerfile itself (root user, missing USER, ADD of remote URLs...) trivy config Dockerfile ``` Accepted risks go in a `.trivyignore` file in the repository, one ID per line, with a comment saying why and until when, so exceptions are reviewed in pull requests and revisited. In CI, run the scan in the same job that builds the image, *before* it's pushed, and again on a schedule against images already in your registry, because new CVEs are published against old images every day. On GitLab the scan is the same command in a job that runs after the build. > [!TIP] > Rebuild images regularly even when your code hasn't changed. Base images receive security patches constantly, and an image built six months ago carries six months of unpatched CVEs. A weekly scheduled rebuild plus an update bot for the `FROM` digest keeps you current. ## Runtime hardening A good image can still be run badly. The runtime settings below apply equally to `docker run`, Docker Compose and Kubernetes `securityContext` (covered in the next lesson). **Drop capabilities.** Linux splits root's power into about 40 capabilities. Docker grants a default set of 14 (including `CHOWN`, `SETUID`, `NET_RAW` and `DAC_OVERRIDE`). You can see them in a process's effective capability mask: ```bash docker run --rm alpine grep CapEff /proc/self/status docker run --rm --cap-drop ALL alpine grep CapEff /proc/self/status ``` ```text CapEff: 00000000a80425fb CapEff: 0000000000000000 ``` Most web services need none. Drop all, and add back only what's proven necessary, such as `NET_BIND_SERVICE` if you really must listen on port 80. **Read-only root filesystem.** If the app can't write to its own filesystem, an attacker can't drop a binary or modify code. Give it a small in-memory `tmpfs` for scratch space: ```bash docker run --rm --read-only alpine touch /evil ``` ```text touch: /evil: Read-only file system ``` **No privilege escalation.** `--security-opt no-new-privileges` stops setuid binaries (like `sudo` or `su`) from granting more privileges than the process started with. **Resource limits.** Without them, one compromised or buggy container can exhaust the host's memory, CPU or process table (a fork bomb). Set memory, CPU and PID limits. Putting it together: ```bash docker run -d --name orders \ --user 65532:65532 \ --read-only --tmpfs /tmp:rw,noexec,nosuid,size=64m \ --cap-drop ALL \ --security-opt no-new-privileges \ --memory 512m --cpus 1 --pids-limit 200 \ -p 8080:8080 \ orders-api:1.4.2 ``` The same in Compose: ```yaml services: orders: image: orders-api:1.4.2 user: "65532:65532" read_only: true tmpfs: - /tmp:rw,noexec,nosuid,size=64m cap_drop: [ALL] security_opt: ["no-new-privileges:true"] mem_limit: 512m cpus: 1 pids_limit: 200 ports: ["8080:8080"] ``` > [!WARNING] > Some settings undo all of this at once. `--privileged` grants every capability and access to host devices. Mounting `/var/run/docker.sock` gives the container full control of the Docker daemon, which is root on the host. `--pid=host` and `--network=host` remove namespace isolation. Treat any of these in a review as a request for root on the machine. Docker's default **seccomp** profile, which blocks around 44 rarely needed and dangerous system calls, is on unless someone passes `--security-opt seccomp=unconfined`; never disable it to "fix" an error without understanding the call being blocked. For defence in depth, rootless Docker or Podman run the daemon itself without root, so even a full escape lands as an unprivileged user. ## Practice Here is a Dockerfile and run command for a small Python API. List every problem, then rewrite both so the image is minimal, non-root and free of secrets, and the container runs with a read-only filesystem, no capabilities and limits. Finally, scan your image with Trivy so it fails on fixable HIGH or CRITICAL vulnerabilities. ```dockerfile FROM python:3.13 ARG PYPI_TOKEN COPY . /app WORKDIR /app RUN pip install --index-url https://__token__:${PYPI_TOKEN}@pypi.acme.internal/simple -r requirements.txt CMD python -m orders ``` ```bash docker run -d -p 8080:8080 --privileged orders-api ``` **Solution** Problems: the full `python:3.13` image (compilers and hundreds of packages); `PYPI_TOKEN` as a build argument, recorded in image metadata and visible in `docker history`; `COPY . /app` copies everything including `.git` and `.env`, and before dependency install, so every code change reinstalls dependencies; running as root; shell-form `CMD`, so `/bin/sh` is PID 1 and doesn't forward signals; and `--privileged` at runtime. ```dockerfile # syntax=docker/dockerfile:1 FROM python:3.13-slim AS build WORKDIR /app RUN python -m venv /venv COPY requirements.txt . RUN --mount=type=secret,id=pip_index_url \ /venv/bin/pip install --no-cache-dir \ --index-url "$(cat /run/secrets/pip_index_url)" -r requirements.txt COPY orders ./orders FROM python:3.13-slim RUN groupadd --system --gid 10001 app \ && useradd --system --uid 10001 --gid app --no-create-home app COPY --from=build /venv /venv COPY --from=build --chown=root:root /app/orders /app/orders WORKDIR /app ENV PATH="/venv/bin:$PATH" PYTHONDONTWRITEBYTECODE=1 USER 10001:10001 EXPOSE 8080 CMD ["python", "-m", "orders"] ``` Add a `.dockerignore` containing `.git`, `.env`, `__pycache__` and `tests`. The virtual environment is copied whole, so it must live at the same path (`/venv`) in both stages. `PYTHONDONTWRITEBYTECODE=1` stops Python trying to write `.pyc` files into the read-only filesystem. ```bash docker build --secret id=pip_index_url,env=PIP_INDEX_URL -t orders-api:1.0.0 . trivy image --exit-code 1 --severity HIGH,CRITICAL --ignore-unfixed orders-api:1.0.0 docker run -d --name orders -p 8080:8080 \ --read-only --tmpfs /tmp:rw,noexec,nosuid,size=64m \ --cap-drop ALL --security-opt no-new-privileges \ --memory 256m --cpus 0.5 --pids-limit 100 \ orders-api:1.0.0 ``` Check: `docker history orders-api:1.0.0` shows no token, `docker exec orders id` prints `uid=10001(app)`, and `docker exec orders touch /app/x` fails with "Read-only file system". A distroless Python base would remove the shell too, at the cost of these easy checks. ## Key takeaways - Containers share the host kernel; every privilege and tool you leave in is something an attacker can use after exploiting your app. - Use multi-stage builds so compilers, dev dependencies and source never reach the final image, and pass build secrets with BuildKit secret mounts, never `ARG`. - Prefer distroless or similarly minimal bases, and run as a numeric non-root UID with code owned by root. - Scan the built image with Trivy, fail on fixable HIGH/CRITICAL findings, record exceptions in `.trivyignore`, and rebuild regularly. - Run with `--cap-drop ALL`, a read-only root filesystem plus `tmpfs`, `no-new-privileges` and memory, CPU and PID limits. - Treat `--privileged`, a mounted Docker socket and host namespaces as root on the host. ## Quiz 1. Why should a registry token used during `npm ci` be passed with `RUN --mount=type=secret` rather than `ARG`? A. ARG values are recorded in the image's build history and metadata, where anyone with the image can read them B. ARG values are not available during RUN steps C. Secret mounts make the build faster D. npm refuses to read tokens from environment variables **Answer: A.** Build arguments end up in image metadata visible through docker history. A secret mount exists only for the duration of that RUN step and is never written to a layer. 2. Trivy reports a HIGH vulnerability with status 'affected' and no fixed version. What does `--ignore-unfixed` do with it? A. Marks it as a false positive permanently B. Hides it from the results, because no upgrade can fix it yet C. Upgrades the package automatically D. Fails the build regardless of severity **Answer: B.** Unfixed vulnerabilities can't be resolved by upgrading, so failing builds on them only causes noise. Track them separately and mitigate in other ways if they matter. 3. A container runs with `--cap-drop ALL`, `--read-only` and a non-root user. A colleague adds `-v /var/run/docker.sock:/var/run/docker.sock` for a monitoring feature. What is the effect? A. None, because capabilities are dropped B. The container can read Docker logs but nothing else C. The container can now control the Docker daemon, which can start privileged containers: effectively root on the host D. The read-only filesystem blocks access to the socket **Answer: C.** The Docker API can launch a new privileged container with the host filesystem mounted. Socket access bypasses every restriction applied to the original container. 4. Why use a numeric `USER 10001:10001` rather than `USER app`? A. Docker cannot resolve user names B. Numeric UIDs are faster at runtime C. Names are only allowed in the build stage D. Kubernetes' runAsNonRoot check can verify a number is not 0, but can't resolve a name without trusting the image's /etc/passwd **Answer: D.** With a named user, the kubelet can't confirm the UID isn't 0 and refuses to start the pod under runAsNonRoot. A numeric UID is unambiguous.