# Linux essentials
> Read and set file permissions, manage users and sudo safely, inspect processes, and run a hardened service with systemd and journalctl.
DevSecOps Engineer roadmap · Foundations · 18 min read · https://levelcode.org/roadmaps/devsecops/linux-essentials
Almost everything you'll secure in DevSecOps runs on Linux: CI runners, containers, Kubernetes nodes, cloud VMs. When a scanner reports "world-writable file" or "container runs as root", or when you investigate a suspicious process at 2 a.m., you need to read the system fluently. This lesson covers the parts that matter most for security: users and groups, permissions, sudo, processes, systemd services and logs. Examples come from Ubuntu 24.04, but everything applies to any modern distribution.
## Users, groups and identity
Every process runs as a **user**, identified by a numeric UID, and belongs to one or more **groups** (GIDs). The kernel only cares about the numbers; names are looked up in `/etc/passwd` and `/etc/group`. UID 0 is `root`, which bypasses almost all permission checks.
```text
$ id
uid=1000(maria) gid=1000(maria) groups=1000(maria),27(sudo),1001(deployers)
$ getent passwd maria webapp
maria:x:1000:1000:Maria Lopez:/home/maria:/bin/bash
webapp:x:996:995::/home/webapp:/usr/sbin/nologin
```
The seven colon-separated fields are: name, password placeholder (`x` means "see `/etc/shadow`"), UID, primary GID, comment, home directory and login shell. Password hashes live in `/etc/shadow`, which only root can read.
Notice `webapp`: a **system account** with a UID below 1000 and the shell `/usr/sbin/nologin`, so nobody can log in as it interactively. Every service should run as its own account like this. If the service is compromised, the attacker gets that account's limited rights, not root's, and not another service's.
```text
$ sudo useradd --system --no-create-home --shell /usr/sbin/nologin webapp
$ sudo groupadd deployers
$ sudo usermod -aG deployers maria
```
The `-a` in `usermod -aG` means *append*. Without it, `-G` replaces the user's supplementary groups, silently removing them from `sudo` and anything else. Group changes take effect at the user's next login.
## Reading and setting permissions
Run `ls -l` and the first column describes the file's type and permissions:
```text
$ ls -l /etc/webapp/
total 8
-rw-r----- 1 root webapp 412 Oct 11 09:35 webapp.env
-rwxr-xr-x 1 root root 1874 Oct 11 09:31 healthcheck.sh
```
Break `-rw-r-----` into four parts:
```text
- rw- r-- ---
type owner group other
```
Each triplet grants **r**ead, **w**rite and e**x**ecute to the file's owner (`root`), its group (`webapp`), and everyone else. So `webapp.env` can be read and written by root, read by members of the `webapp` group (the service), and not touched by anyone else. That's exactly what you want for a file holding database credentials.
### Octal notation
Each permission is a bit: read = 4, write = 2, execute = 1. Add them per triplet to get one digit, and write three digits for owner, group and other:
| Symbolic | Octal | Typical use |
|---|---|---|
| `rw-------` | 600 | SSH private keys, secrets only the owner reads |
| `rw-r-----` | 640 | Config readable by a service's group |
| `rw-r--r--` | 644 | Ordinary public files |
| `rwx------` | 700 | Private directories, e.g. `~/.ssh` |
| `rwxr-xr-x` | 755 | Programs and directories everyone may use |
```text
$ chmod 640 /etc/webapp/webapp.env
$ stat -c '%a %A %U:%G %n' /etc/webapp/webapp.env
640 -rw-r----- root:webapp /etc/webapp/webapp.env
$ chmod u+x,g-r notes.txt # symbolic form: add or remove individual bits
$ sudo chown root:webapp /etc/webapp/webapp.env
```
For **directories** the bits mean something different: `r` lists names, `w` creates and deletes entries, and `x` lets you enter the directory and access files inside it. A file with mode 644 in a directory with mode 700 is still unreachable for other users, because they can't traverse the directory.
Use the calculator below to connect the three notations. Toggle bits in the grid, or type an octal mode or an `ls -l` string, and read the plain-English summary. Switch to **directory** to see how the meaning of each bit changes; the bottom row is the optional fourth digit covered in the next section.
> **Interactive: chmod calculator.** Each digit is the sum of r=4, w=2 and x=1 for one class. On a directory, x is what lets you enter it and reach anything inside, so r or w without x is almost never what you want. Try it: https://levelcode.org/labs/chmod-calculator
### Defaults and special bits
New files get their mode from the **umask**, a mask of bits to remove. The usual default `022` gives files 644 and directories 755. A service that writes sensitive files should use `027` or `077`:
```text
$ umask 027
$ touch report.csv && mkdir exports
$ ls -ld report.csv exports
-rw-r----- 1 maria maria 0 Oct 11 09:35 report.csv
drwxr-x--- 2 maria maria 4096 Oct 11 09:35 exports
```
Three special bits form an optional fourth (leading) octal digit:
- **setuid (4)**: a program runs with its *owner's* privileges. `/usr/bin/passwd` is setuid root (`-rwsr-xr-x`, mode 4755) so ordinary users can update `/etc/shadow`. Unexpected setuid binaries are a classic privilege-escalation path.
- **setgid (2)**: on a directory, new files inherit the directory's group. Useful for shared project directories.
- **sticky (1)**: in a writable directory, only a file's owner (or the directory's owner, or root) can delete or rename it. `/tmp` is `drwxrwxrwt` (mode 1777).
Auditing for risky permissions is a common hardening task:
```text
$ sudo find / -xdev -perm -4000 -type f 2>/dev/null # setuid files
/usr/bin/passwd
/usr/bin/sudo
/usr/bin/su
...
$ sudo find /etc /opt -xdev -perm -0002 -type f # world-writable files
```
The `-perm -4000` form means "at least these bits set". Compare the setuid list against a known-good baseline; anything new deserves an explanation.
> [!WARNING]
> `chmod 777` is never the fix for "permission denied". It lets every user and every compromised process modify the file. Find out which user needs access and grant exactly that, usually via group ownership and mode 640 or 660.
## sudo: borrowing privileges carefully
Logging in as root directly leaves no record of who did what. `sudo` runs a single command as another user (root by default), checks the caller against the rules in `/etc/sudoers`, and logs every use. Always edit sudo rules with `visudo`, which refuses to save a file with syntax errors that would lock everyone out.
Rather than giving a deploy group full root, allow only the command it needs. Put the rule in its own file under `/etc/sudoers.d/`:
```text
# /etc/sudoers.d/deployers (edit with: sudo visudo -f /etc/sudoers.d/deployers)
%deployers ALL=(root) /usr/bin/systemctl restart webapp.service
```
Read it as: members of group `deployers`, on any host, may run as root exactly `systemctl restart webapp.service`. Check what a user is allowed to do:
```text
$ sudo -l -U maria
User maria may run the following commands on web-01:
(root) /usr/bin/systemctl restart webapp.service
$ sudo systemctl stop webapp.service
Sorry, user maria is not allowed to execute '/usr/bin/systemctl stop webapp.service' as root on web-01.
```
Be careful what you allow. Rules that permit editors (`vim`), pagers (`less`), interpreters (`python3`) or `ALL` with a wildcard effectively grant a full root shell, because those programs can spawn shells. `NOPASSWD:` removes the password prompt; reserve it for automation accounts with tightly scoped commands.
## Processes and signals
A **process** is a running program with a PID, a parent (PPID), a user and a state. `ps` gives you a snapshot; `top` or `htop` give a live view.
```text
$ ps -eo user,pid,ppid,stat,%cpu,%mem,cmd --sort=-%cpu | head -5
USER PID PPID STAT %CPU %MEM CMD
webapp 1842 1 Ss 3.1 2.4 /opt/webapp/venv/bin/python3 /opt/webapp/venv/bin/gunicorn --bind 127.0.0.1:8000 app:app
webapp 1851 1842 S 1.2 2.1 /opt/webapp/venv/bin/python3 /opt/webapp/venv/bin/gunicorn --bind 127.0.0.1:8000 app:app
www-data 977 975 S 0.4 0.3 nginx: worker process
root 975 1 Ss 0.0 0.1 nginx: master process /usr/sbin/nginx
```
Security questions to ask of that list: is anything running as root that shouldn't be? Is a process listening on `0.0.0.0` that should be bound to `127.0.0.1`? (You'll check listening sockets with `ss` in the networking lesson.) Is there a process you don't recognise, perhaps with a random name in `/tmp`?
Every process also has a directory in `/proc`. Two useful entries:
```text
$ sudo ls -l /proc/1842/exe
lrwxrwxrwx 1 webapp webapp 0 Oct 11 10:02 /proc/1842/exe -> /usr/bin/python3.12
$ sudo cat /proc/1842/environ | tr '\0' '\n' | cut -d= -f1
PATH
DATABASE_URL
LANG
```
`exe` shows which binary is actually running, even if the file on disk was deleted (a common trick in malware). `environ` contains the process's environment variables, which is why secrets in environment variables are readable by root and by the same user. Here only the names are printed; the values would include the database password.
To stop a process, send it a **signal** with `kill`. `SIGTERM` (15, the default) asks it to shut down cleanly; `SIGKILL` (9) ends it immediately without cleanup. Try TERM first:
```text
$ sudo kill 1851 # SIGTERM (sudo: the process belongs to webapp, not you)
$ sudo kill -9 1851 # SIGKILL, only if TERM didn't work
```
## Services with systemd
On modern distributions, **systemd** starts and supervises long-running services. Each service is described by a **unit file**. Here is a production-quality unit for the web app, including sandboxing options that systemd applies before your code runs:
```ini
# /etc/systemd/system/webapp.service
[Unit]
Description=Acme web app
After=network-online.target
Wants=network-online.target
[Service]
User=webapp
Group=webapp
WorkingDirectory=/opt/webapp
EnvironmentFile=/etc/webapp/webapp.env
ExecStart=/opt/webapp/venv/bin/gunicorn --bind 127.0.0.1:8000 app:app
Restart=on-failure
RestartSec=5
# Hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ReadWritePaths=/var/lib/webapp
CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
SystemCallFilter=@system-service
[Install]
WantedBy=multi-user.target
```
What the hardening lines do:
- `User=`/`Group=` run the app as the unprivileged system account.
- `NoNewPrivileges=yes` stops the process and its children gaining privileges, so setuid binaries don't help an attacker.
- `ProtectSystem=strict` mounts the whole filesystem read-only for this service, except paths in `ReadWritePaths=`. `ProtectHome=yes` hides `/home`, `/root` and `/run/user`.
- `PrivateTmp=yes` gives the service its own `/tmp`, so it can't read other services' temporary files.
- `CapabilityBoundingSet=` (empty) drops all Linux capabilities, the fine-grained slices of root power.
- `SystemCallFilter=@system-service` blocks system calls an ordinary service never needs.
Load and manage it with `systemctl`:
```text
$ sudo systemctl daemon-reload # re-read unit files after editing
$ sudo systemctl enable --now webapp # start now and at every boot
$ systemctl status webapp
● webapp.service - Acme web app
Loaded: loaded (/etc/systemd/system/webapp.service; enabled; preset: enabled)
Active: active (running) since Sun 2026-10-11 10:02:14 UTC; 8s ago
Main PID: 1842 (gunicorn)
Tasks: 3 (limit: 4613)
Memory: 41.2M (peak: 41.5M)
CPU: 612ms
CGroup: /system.slice/webapp.service
├─1842 /opt/webapp/venv/bin/python3 /opt/webapp/venv/bin/gunicorn --bind 127.0.0.1:8000 app:app
└─1851 /opt/webapp/venv/bin/python3 /opt/webapp/venv/bin/gunicorn --bind 127.0.0.1:8000 app:app
```
`systemd-analyze security webapp` scores a unit's sandboxing from 0 (locked down) to 10 (fully exposed) and lists each setting it checks. Running it on an unhardened unit is an eye-opening way to see how much access a default service has.
## Logs with journalctl
systemd collects the output of every service, plus kernel and authentication messages, in the **journal**. `journalctl` queries it:
```text
$ journalctl -u webapp --since "10 min ago"
Oct 11 10:02:14 web-01 systemd[1]: Started webapp.service - Acme web app.
Oct 11 10:02:15 web-01 gunicorn[1842]: [INFO] Listening at: http://127.0.0.1:8000 (1842)
Oct 11 10:05:41 web-01 gunicorn[1851]: [ERROR] could not connect to database: password authentication failed
$ journalctl -u webapp -p err -b # errors and worse, since this boot
$ journalctl -u webapp -f # follow live, like tail -f
```
For security investigations, the SSH daemon's log is often the first stop. Failed logins look like this:
```text
$ journalctl -u ssh --since today | grep "Failed password"
Oct 11 03:12:07 web-01 sshd[2210]: Failed password for invalid user admin from 203.0.113.45 port 51122 ssh2
Oct 11 03:12:09 web-01 sshd[2210]: Failed password for invalid user admin from 203.0.113.45 port 51122 ssh2
```
Thousands of these from internet addresses are normal background noise on any public server, which is why SSH should accept only keys. In `/etc/ssh/sshd_config` (or a file in `/etc/ssh/sshd_config.d/`), set `PasswordAuthentication no` and `PermitRootLogin no`, check the syntax with `sudo sshd -t`, then reload with `sudo systemctl reload ssh`. Keep your current session open until you've confirmed a new key-based login works.
> [!TIP]
> Reading the journal requires membership of the `adm` or `systemd-journal` group, or sudo. Logs often contain sensitive data, so don't grant that group casually.
## Practice
On a Linux VM or container you control, deploy a tiny service the secure way:
1. Create a system user `hello` with no login shell and no home directory.
2. Create `/opt/hello/server.sh` that runs `exec python3 -m http.server 8080 --bind 127.0.0.1 --directory /opt/hello/www` and an `index.html` in `/opt/hello/www`. Make the script owned by root, executable, and not writable by `hello`.
3. Write `/etc/systemd/system/hello.service` that runs it as `hello` with `NoNewPrivileges`, `ProtectSystem=strict`, `ProtectHome` and `PrivateTmp`.
4. Start it, confirm with `curl -s localhost:8080`, find its PID and user with `ps`, and read its logs with `journalctl`.
5. Grant a group `hello-ops` permission to restart (only restart) the service with sudo.
**Solution**
```bash
sudo useradd --system --no-create-home --shell /usr/sbin/nologin hello
sudo mkdir -p /opt/hello/www
echo 'hello
' | sudo tee /opt/hello/www/index.html >/dev/null
sudo tee /opt/hello/server.sh >/dev/null <<'EOF'
#!/bin/sh
exec python3 -m http.server 8080 --bind 127.0.0.1 --directory /opt/hello/www
EOF
sudo chown -R root:root /opt/hello
sudo chmod 755 /opt/hello/server.sh # root writes; everyone (incl. hello) reads and runs
sudo chmod 644 /opt/hello/www/index.html
sudo tee /etc/systemd/system/hello.service >/dev/null <<'EOF'
[Unit]
Description=Hello static server
[Service]
User=hello
Group=hello
ExecStart=/opt/hello/server.sh
Restart=on-failure
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now hello
curl -s localhost:8080 # hello
ps -o user,pid,cmd -C python3 # USER is hello
journalctl -u hello -n 5
sudo groupadd hello-ops
echo '%hello-ops ALL=(root) /usr/bin/systemctl restart hello.service' | sudo tee /etc/sudoers.d/hello-ops >/dev/null
sudo chmod 440 /etc/sudoers.d/hello-ops
sudo visudo -cf /etc/sudoers.d/hello-ops # prints: parsed OK
```
`exec` replaces the shell with Python, so systemd supervises Python directly and signals reach it. `hello` can run the script but not modify it, so a compromise of the server process can't rewrite what systemd executes on the next restart.
## Key takeaways
- Run every service as its own unprivileged system account with a `nologin` shell; UID 0 bypasses nearly all checks.
- Read permissions as owner/group/other triplets; in octal, r=4, w=2, x=1. Secrets are typically 600 or 640, never 777.
- Audit setuid binaries and world-writable files; set a restrictive umask for services that write sensitive data.
- Use `usermod -aG` (append) and scope sudo rules to exact commands, edited with `visudo`.
- `ps` and `/proc/` show who is running what; environment variables of a process are readable by its user and root.
- systemd units can sandbox services (`NoNewPrivileges`, `ProtectSystem=strict`, `PrivateTmp`), and `journalctl -u` is your first stop when investigating.
## Quiz
1. A file has mode 640, owner root and group webapp. Who can read it?
A. Everyone on the system
B. Only root
C. root and members of the webapp group
D. Members of the webapp group, but not root
**Answer: C.** 6 (rw-) for the owner root, 4 (r--) for group webapp, 0 for others. Root could also read it regardless, since UID 0 bypasses permission checks.
2. What is the effect of running `sudo usermod -G docker maria` (without -a)?
A. maria is added to docker and keeps her other groups
B. maria's supplementary groups are replaced with just docker, removing her from groups like sudo
C. The docker group is deleted
D. maria's primary group becomes docker and nothing else changes
**Answer: B.** -G sets the full list of supplementary groups. -a (append) is required to add one group without removing the others.
3. Which sudoers rule follows least privilege for a team that only needs to restart the app?
A. %ops ALL=(root) /usr/bin/systemctl restart webapp.service
B. %ops ALL=(ALL) ALL
C. %ops ALL=(root) NOPASSWD: /usr/bin/vim /etc/systemd/system/webapp.service
D. %ops ALL=(root) /usr/bin/python3
**Answer: A.** It allows exactly one command. Granting vim or python3 as root is effectively a root shell, because both can spawn a shell; ALL is unrestricted.
4. What does ProtectSystem=strict do in a systemd service unit?
A. Prevents the service from being stopped
B. Encrypts the service's files on disk
C. Blocks all network access for the service
D. Makes the entire filesystem read-only for the service, except paths listed in ReadWritePaths=
**Answer: D.** strict mounts the whole file hierarchy read-only for that service. Even if an attacker controls the process, they can't modify binaries or config outside the allowed writable paths.