Linux essentials

Almost everything you'll secure in DevSecOps runs on Linux: CI runners, containers, Kubernetes nodes, cloud VMs. When a scanner reports "world-writable file" or "container runs as root", or when you investigate a suspicious process at 2 a.m., you need to read the system fluently. This lesson covers the parts that matter most for security: users and groups, permissions, sudo, processes, systemd services and logs. Examples come from Ubuntu 24.04, but everything applies to any modern distribution.

#Users, groups and identity

Every process runs as a user, identified by a numeric UID, and belongs to one or more groups (GIDs). The kernel only cares about the numbers; names are looked up in /etc/passwd and /etc/group. UID 0 is root, which bypasses almost all permission checks.

$ id
uid=1000(maria) gid=1000(maria) groups=1000(maria),27(sudo),1001(deployers)

$ getent passwd maria webapp
maria:x:1000:1000:Maria Lopez:/home/maria:/bin/bash
webapp:x:996:995::/home/webapp:/usr/sbin/nologin

The seven colon-separated fields are: name, password placeholder (x means "see /etc/shadow"), UID, primary GID, comment, home directory and login shell. Password hashes live in /etc/shadow, which only root can read.

Notice webapp: a system account with a UID below 1000 and the shell /usr/sbin/nologin, so nobody can log in as it interactively. Every service should run as its own account like this. If the service is compromised, the attacker gets that account's limited rights, not root's, and not another service's.

$ sudo useradd --system --no-create-home --shell /usr/sbin/nologin webapp
$ sudo groupadd deployers
$ sudo usermod -aG deployers maria

The -a in usermod -aG means append. Without it, -G replaces the user's supplementary groups, silently removing them from sudo and anything else. Group changes take effect at the user's next login.

#Reading and setting permissions

Run ls -l and the first column describes the file's type and permissions:

$ ls -l /etc/webapp/
total 8
-rw-r----- 1 root webapp  412 Oct 11 09:35 webapp.env
-rwxr-xr-x 1 root root   1874 Oct 11 09:31 healthcheck.sh

Break -rw-r----- into four parts:

-    rw-    r--    ---
type owner  group  other

Each triplet grants read, write and execute to the file's owner (root), its group (webapp), and everyone else. So webapp.env can be read and written by root, read by members of the webapp group (the service), and not touched by anyone else. That's exactly what you want for a file holding database credentials.

#Octal notation

Each permission is a bit: read = 4, write = 2, execute = 1. Add them per triplet to get one digit, and write three digits for owner, group and other:

$ chmod 640 /etc/webapp/webapp.env
$ stat -c '%a %A %U:%G %n' /etc/webapp/webapp.env
640 -rw-r----- root:webapp /etc/webapp/webapp.env

$ chmod u+x,g-r notes.txt      # symbolic form: add or remove individual bits
$ sudo chown root:webapp /etc/webapp/webapp.env

For directories the bits mean something different: r lists names, w creates and deletes entries, and x lets you enter the directory and access files inside it. A file with mode 644 in a directory with mode 700 is still unreachable for other users, because they can't traverse the directory.

Use the calculator below to connect the three notations. Toggle bits in the grid, or type an octal mode or an ls -l string, and read the plain-English summary. Switch to directory to see how the meaning of each bit changes; the bottom row is the optional fourth digit covered in the next section.

#Defaults and special bits

New files get their mode from the umask, a mask of bits to remove. The usual default 022 gives files 644 and directories 755. A service that writes sensitive files should use 027 or 077:

$ umask 027
$ touch report.csv && mkdir exports
$ ls -ld report.csv exports
-rw-r----- 1 maria maria    0 Oct 11 09:35 report.csv
drwxr-x--- 2 maria maria 4096 Oct 11 09:35 exports

Three special bits form an optional fourth (leading) octal digit:

• setuid (4): a program runs with its owner's privileges. /usr/bin/passwd is setuid root (-rwsr-xr-x, mode 4755) so ordinary users can update /etc/shadow. Unexpected setuid binaries are a classic privilege-escalation path.
• setgid (2): on a directory, new files inherit the directory's group. Useful for shared project directories.
• sticky (1): in a writable directory, only a file's owner (or the directory's owner, or root) can delete or rename it. /tmp is drwxrwxrwt (mode 1777).

Auditing for risky permissions is a common hardening task:

$ sudo find / -xdev -perm -4000 -type f 2>/dev/null     # setuid files
/usr/bin/passwd
/usr/bin/sudo
/usr/bin/su
...
$ sudo find /etc /opt -xdev -perm -0002 -type f         # world-writable files

The -perm -4000 form means "at least these bits set". Compare the setuid list against a known-good baseline; anything new deserves an explanation.

#sudo: borrowing privileges carefully

Logging in as root directly leaves no record of who did what. sudo runs a single command as another user (root by default), checks the caller against the rules in /etc/sudoers, and logs every use. Always edit sudo rules with visudo, which refuses to save a file with syntax errors that would lock everyone out.

Rather than giving a deploy group full root, allow only the command it needs. Put the rule in its own file under /etc/sudoers.d/:

# /etc/sudoers.d/deployers  (edit with: sudo visudo -f /etc/sudoers.d/deployers)
%deployers ALL=(root) /usr/bin/systemctl restart webapp.service

Read it as: members of group deployers, on any host, may run as root exactly systemctl restart webapp.service. Check what a user is allowed to do:

$ sudo -l -U maria
User maria may run the following commands on web-01:
    (root) /usr/bin/systemctl restart webapp.service

$ sudo systemctl stop webapp.service
Sorry, user maria is not allowed to execute '/usr/bin/systemctl stop webapp.service' as root on web-01.

Be careful what you allow. Rules that permit editors (vim), pagers (less), interpreters (python3) or ALL with a wildcard effectively grant a full root shell, because those programs can spawn shells. NOPASSWD: removes the password prompt; reserve it for automation accounts with tightly scoped commands.

#Processes and signals

A process is a running program with a PID, a parent (PPID), a user and a state. ps gives you a snapshot; top or htop give a live view.

$ ps -eo user,pid,ppid,stat,%cpu,%mem,cmd --sort=-%cpu | head -5
USER       PID    PPID STAT %CPU %MEM CMD
webapp    1842       1 Ss    3.1  2.4 /opt/webapp/venv/bin/python3 /opt/webapp/venv/bin/gunicorn --bind 127.0.0.1:8000 app:app
webapp    1851    1842 S     1.2  2.1 /opt/webapp/venv/bin/python3 /opt/webapp/venv/bin/gunicorn --bind 127.0.0.1:8000 app:app
www-data   977     975 S     0.4  0.3 nginx: worker process
root       975       1 Ss    0.0  0.1 nginx: master process /usr/sbin/nginx

Security questions to ask of that list: is anything running as root that shouldn't be? Is a process listening on 0.0.0.0 that should be bound to 127.0.0.1? (You'll check listening sockets with ss in the networking lesson.) Is there a process you don't recognise, perhaps with a random name in /tmp?

Every process also has a directory in /proc. Two useful entries:

$ sudo ls -l /proc/1842/exe
lrwxrwxrwx 1 webapp webapp 0 Oct 11 10:02 /proc/1842/exe -> /usr/bin/python3.12
$ sudo cat /proc/1842/environ | tr '\0' '\n' | cut -d= -f1
PATH
DATABASE_URL
LANG

exe shows which binary is actually running, even if the file on disk was deleted (a common trick in malware). environ contains the process's environment variables, which is why secrets in environment variables are readable by root and by the same user. Here only the names are printed; the values would include the database password.

To stop a process, send it a signal with kill. SIGTERM (15, the default) asks it to shut down cleanly; SIGKILL (9) ends it immediately without cleanup. Try TERM first:

$ sudo kill 1851          # SIGTERM (sudo: the process belongs to webapp, not you)
$ sudo kill -9 1851       # SIGKILL, only if TERM didn't work

#Services with systemd

On modern distributions, systemd starts and supervises long-running services. Each service is described by a unit file. Here is a production-quality unit for the web app, including sandboxing options that systemd applies before your code runs:

# /etc/systemd/system/webapp.service
[Unit]
Description=Acme web app
After=network-online.target
Wants=network-online.target

[Service]
User=webapp
Group=webapp
WorkingDirectory=/opt/webapp
EnvironmentFile=/etc/webapp/webapp.env
ExecStart=/opt/webapp/venv/bin/gunicorn --bind 127.0.0.1:8000 app:app
Restart=on-failure
RestartSec=5

# Hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ReadWritePaths=/var/lib/webapp
CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
SystemCallFilter=@system-service

[Install]
WantedBy=multi-user.target

What the hardening lines do:

• User=/Group= run the app as the unprivileged system account.
• NoNewPrivileges=yes stops the process and its children gaining privileges, so setuid binaries don't help an attacker.
• ProtectSystem=strict mounts the whole filesystem read-only for this service, except paths in ReadWritePaths=. ProtectHome=yes hides /home, /root and /run/user.
• PrivateTmp=yes gives the service its own /tmp, so it can't read other services' temporary files.
• CapabilityBoundingSet= (empty) drops all Linux capabilities, the fine-grained slices of root power.
• SystemCallFilter=@system-service blocks system calls an ordinary service never needs.

Load and manage it with systemctl:

$ sudo systemctl daemon-reload            # re-read unit files after editing
$ sudo systemctl enable --now webapp      # start now and at every boot
$ systemctl status webapp
● webapp.service - Acme web app
     Loaded: loaded (/etc/systemd/system/webapp.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-10-11 10:02:14 UTC; 8s ago
   Main PID: 1842 (gunicorn)
      Tasks: 3 (limit: 4613)
     Memory: 41.2M (peak: 41.5M)
        CPU: 612ms
     CGroup: /system.slice/webapp.service
             ├─1842 /opt/webapp/venv/bin/python3 /opt/webapp/venv/bin/gunicorn --bind 127.0.0.1:8000 app:app
             └─1851 /opt/webapp/venv/bin/python3 /opt/webapp/venv/bin/gunicorn --bind 127.0.0.1:8000 app:app

systemd-analyze security webapp scores a unit's sandboxing from 0 (locked down) to 10 (fully exposed) and lists each setting it checks. Running it on an unhardened unit is an eye-opening way to see how much access a default service has.

#Logs with journalctl

systemd collects the output of every service, plus kernel and authentication messages, in the journal. journalctl queries it:

$ journalctl -u webapp --since "10 min ago"
Oct 11 10:02:14 web-01 systemd[1]: Started webapp.service - Acme web app.
Oct 11 10:02:15 web-01 gunicorn[1842]: [INFO] Listening at: http://127.0.0.1:8000 (1842)
Oct 11 10:05:41 web-01 gunicorn[1851]: [ERROR] could not connect to database: password authentication failed

$ journalctl -u webapp -p err -b      # errors and worse, since this boot
$ journalctl -u webapp -f             # follow live, like tail -f

For security investigations, the SSH daemon's log is often the first stop. Failed logins look like this:

$ journalctl -u ssh --since today | grep "Failed password"
Oct 11 03:12:07 web-01 sshd[2210]: Failed password for invalid user admin from 203.0.113.45 port 51122 ssh2
Oct 11 03:12:09 web-01 sshd[2210]: Failed password for invalid user admin from 203.0.113.45 port 51122 ssh2

Thousands of these from internet addresses are normal background noise on any public server, which is why SSH should accept only keys. In /etc/ssh/sshd_config (or a file in /etc/ssh/sshd_config.d/), set PasswordAuthentication no and PermitRootLogin no, check the syntax with sudo sshd -t, then reload with sudo systemctl reload ssh. Keep your current session open until you've confirmed a new key-based login works.

#Practice

On a Linux VM or container you control, deploy a tiny service the secure way:

1. Create a system user hello with no login shell and no home directory.
2. Create /opt/hello/server.sh that runs exec python3 -m http.server 8080 --bind 127.0.0.1 --directory /opt/hello/www and an index.html in /opt/hello/www. Make the script owned by root, executable, and not writable by hello.
3. Write /etc/systemd/system/hello.service that runs it as hello with NoNewPrivileges, ProtectSystem=strict, ProtectHome and PrivateTmp.
4. Start it, confirm with curl -s localhost:8080, find its PID and user with ps, and read its logs with journalctl.
5. Grant a group hello-ops permission to restart (only restart) the service with sudo.

sudo useradd --system --no-create-home --shell /usr/sbin/nologin hello
sudo mkdir -p /opt/hello/www
echo '<h1>hello</h1>' | sudo tee /opt/hello/www/index.html >/dev/null
sudo tee /opt/hello/server.sh >/dev/null <<'EOF'
#!/bin/sh
exec python3 -m http.server 8080 --bind 127.0.0.1 --directory /opt/hello/www
EOF
sudo chown -R root:root /opt/hello
sudo chmod 755 /opt/hello/server.sh          # root writes; everyone (incl. hello) reads and runs
sudo chmod 644 /opt/hello/www/index.html

sudo tee /etc/systemd/system/hello.service >/dev/null <<'EOF'
[Unit]
Description=Hello static server

[Service]
User=hello
Group=hello
ExecStart=/opt/hello/server.sh
Restart=on-failure
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now hello
curl -s localhost:8080            # <h1>hello</h1>
ps -o user,pid,cmd -C python3     # USER is hello
journalctl -u hello -n 5

sudo groupadd hello-ops
echo '%hello-ops ALL=(root) /usr/bin/systemctl restart hello.service' | sudo tee /etc/sudoers.d/hello-ops >/dev/null
sudo chmod 440 /etc/sudoers.d/hello-ops
sudo visudo -cf /etc/sudoers.d/hello-ops      # prints: parsed OK

#Key takeaways

• Run every service as its own unprivileged system account with a nologin shell; UID 0 bypasses nearly all checks.
• Read permissions as owner/group/other triplets; in octal, r=4, w=2, x=1. Secrets are typically 600 or 640, never 777.
• Audit setuid binaries and world-writable files; set a restrictive umask for services that write sensitive data.
• Use usermod -aG (append) and scope sudo rules to exact commands, edited with visudo.
• ps and /proc/<pid> show who is running what; environment variables of a process are readable by its user and root.
• systemd units can sandbox services (NoNewPrivileges, ProtectSystem=strict, PrivateTmp), and journalctl -u is your first stop when investigating.