Backup & disaster recovery

Every other lesson in this roadmap tries to stop bad things happening. This one assumes they will: someone runs a DELETE against the wrong database, ransomware encrypts every server it can reach, or a region goes down. Whether that's a bad afternoon or the end of the company depends on whether you can get your data and systems back, how much you lose, and how long it takes. This lesson covers recovery objectives, kinds of backup, point-in-time recovery for PostgreSQL, immutable backups that survive ransomware, key management, restore testing, and disaster recovery strategies.

#RPO and RTO

Two numbers define "recoverable" for each system, and the business should set them:

• Recovery point objective (RPO): the most data, measured in time, you can afford to lose. RPO 15 minutes means at most the last 15 minutes of writes may be gone.
• Recovery time objective (RTO): the longest the service may be down. RTO 2 hours means customers wait at most 2 hours.

Worked example: RPO. Acme's orders database is snapshotted nightly at 02:00. At 17:40 a storage failure corrupts it, and 15 hours 40 minutes of orders are lost. The worst case, a failure just before 02:00, loses almost 24 hours: that is the real RPO of a nightly backup. If the business needs "at most 5 minutes", nightly backups can't deliver it at any price; you need continuous log archiving.

Worked example: RTO. RTO covers every step from the alarm to customers being served, not just copying a file. Budget it with speeds you've measured, here for an 800 GB database:

If the RTO is 1 hour, this design fails, and you've learned it before the disaster rather than during it. Fixes include more frequent base backups (less log to replay) or a running standby. Set objectives per system: the payment ledger and the wiki don't need the same answer, and tighter costs more.

#Kinds of backup

Full, incremental and differential describe how much each backup copies:

Incrementals are cheapest, but every link in the chain must be intact: one corrupt Tuesday incremental breaks every later restore. Deduplicating tools (restic, pgBackRest, Velero's data movers) make each backup logically full while storing only new data. PostgreSQL 17 added native incrementals: pg_basebackup --incremental=<previous backup_manifest> with summarize_wal = on, merged for restore with pg_combinebackup.

How you capture data matters as much:

• Storage snapshots (EBS, RDS, disk snapshots) copy blocks quickly, but they're crash-consistent unless the application is quiesced, and by default sit in the same account and region as the original. A snapshot the attacker can delete with the same credentials isn't a backup yet.
• Logical backups (pg_dump) export schema and rows. They're portable across versions and can restore one table, but are slow for large databases and capture only the moment the dump started.
• Physical backups plus log archiving copy the database files and then every change since, which enables point-in-time recovery.

pg_dump -Fc -d orders -f /backups/logical/orders-2026-10-11.dump
pg_restore -d orders_scratch -t customers /backups/logical/orders-2026-10-11.dump

The custom format (-Fc) is compressed and lets pg_restore pick objects, here just customers into a scratch database.

#Point-in-time recovery for PostgreSQL

PostgreSQL records every change in its write-ahead log (WAL) before changing data files. Keep a base backup and every WAL segment since, and you can replay history and stop at any moment: one second before the mistake. Enable archiving in postgresql.conf:

wal_level = replica
archive_mode = on
archive_command = 'test ! -f /mnt/wal-archive/%f && cp %p /mnt/wal-archive/%f'
archive_timeout = 60

%p is the finished segment's path and %f its name; test ! -f refuses to overwrite. archive_timeout = 60 forces a segment at least every minute, bounding RPO to about a minute. In production, use pgBackRest, WAL-G or Barman, which ship segments to object storage with compression, encryption and verification. Take base backups on a schedule:

pg_basebackup -h db1.internal -U replicator -D /backups/base/2026-10-08 \
  -Ft -z -Xs -P --checkpoint=fast

That writes compressed tar files, streams the WAL the backup needs to be consistent, shows progress, and starts without waiting for a checkpoint. Managed databases do this for you: RDS keeps continuous backups for up to 35 days, and aws rds restore-db-instance-to-point-in-time restores to any second in that window, always to a new instance.

#Worked example: recovering from an accidental DELETE

On 2026-10-08, a clean-up script meant for staging runs against production. The database log shows its DELETE committed at 14:32:07 UTC, removing 31,204 old orders and, through ON DELETE CASCADE, their items. New orders keep arriving.

Rolling the whole database back to 14:32 would also discard every legitimate order since. Instead, restore a separate copy to just before the mistake, pull the deleted rows out, and put them back. Step through the flow and notice that production never stops taking orders.

On a fresh host with PostgreSQL 17 (never the production server), extract the base backup:

sudo -u postgres mkdir -p /srv/restore/pgdata/pg_wal
sudo -u postgres chmod 700 /srv/restore/pgdata
sudo -u postgres tar -xzf /backups/base/2026-10-08/base.tar.gz -C /srv/restore/pgdata
sudo -u postgres tar -xzf /backups/base/2026-10-08/pg_wal.tar.gz -C /srv/restore/pgdata/pg_wal

Add recovery settings to the restored cluster's postgresql.conf. archive_mode = off matters: otherwise the copy would archive its own WAL into the production archive.

restore_command = 'cp /mnt/wal-archive/%f %p'
recovery_target_time = '2026-10-08 14:32:00+00'
recovery_target_action = 'pause'
archive_mode = off
port = 5433

Create the signal file that requests targeted recovery, and start the server:

sudo -u postgres touch /srv/restore/pgdata/recovery.signal
sudo -u postgres pg_ctl -D /srv/restore/pgdata -l /srv/restore/recovery.log start

LOG:  starting point-in-time recovery to 2026-10-08 14:32:00+00
LOG:  restored log file "000000010000003A00000012" from archive
...
LOG:  recovery stopping before commit of transaction 9184431, time 2026-10-08 14:32:00.183+00
LOG:  pausing at the end of recovery
HINT:  Execute pg_wal_replay_resume() to promote.

Recovery stopped before the first commit after the target, an ordinary order at 14:32:00.183. While paused, the server answers read-only queries, which is all you need. Export the deleted rows, then load them into production, parent table first so foreign keys are satisfied:

-- psql -p 5433 -d orders   (the restored copy)
\copy (SELECT * FROM orders WHERE created_at < '2026-10-01') TO 'orders.csv' CSV
\copy (SELECT oi.* FROM order_items oi JOIN orders o ON o.id = oi.order_id WHERE o.created_at < '2026-10-01') TO 'order_items.csv' CSV

-- psql -h db1.internal -d orders   (production)
\copy orders FROM 'orders.csv' CSV
\copy order_items FROM 'order_items.csv' CSV

If the whole database were lost instead, you'd validate the copy, run SELECT pg_wal_replay_resume(); to promote it, and point the application at it.

#3-2-1-1-0 and immutable backups

The classic rule is 3-2-1: three copies, on two kinds of storage, one off-site. Ransomware added two digits: 1 copy offline, air-gapped or immutable, and 0 errors when you verify by restoring.

Ransomware operators target backups first. They often spend days or weeks inside a network beforehand, finding the backup console, deleting snapshots, shortening retention and disabling versioning. Only then do they encrypt production. So design as if the attacker holds production administrator credentials.

Immutability. S3 Object Lock stores objects as WORM (write once, read many): each version has a retention date before which nobody can delete or overwrite it.

aws s3api create-bucket --bucket acme-backups-vault --region eu-west-1 \
  --create-bucket-configuration LocationConstraint=eu-west-1 \
  --object-lock-enabled-for-bucket

aws s3api put-object-lock-configuration --bucket acme-backups-vault \
  --object-lock-configuration '{"ObjectLockEnabled": "Enabled", "Rule": {"DefaultRetention": {"Mode": "COMPLIANCE", "Days": 35}}}'

Enabling Object Lock at creation also turns on versioning, which it requires. Every new version is now locked for 35 days. Governance mode can be bypassed by principals with s3:BypassGovernanceRetention; compliance mode can't be shortened or bypassed by anyone, including the root user. Make retention longer than a likely dwell time plus time to notice. AWS Backup Vault Lock, Azure immutable blob storage and Google Cloud bucket locks offer the same guarantee.

Isolation. Keep the vault in a separate account with its own administrators, reached only through break-glass (the cloud IAM lesson's pattern). Production gets a write-only path in, through the bucket policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ProductionBackupWriterPutOnly",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::111122223333:role/orders-backup-writer" },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::acme-backups-vault/orders/*"
    }
  ]
}

The production role can add backups but not list, read, delete or reconfigure anything, and new buckets enforce bucket-owner ownership, so its objects belong to the vault account. An SCP on the vault account stops even its own administrators weakening the protection outside break-glass:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ProtectBackupVault",
      "Effect": "Deny",
      "Action": [
        "s3:DeleteBucket",
        "s3:PutBucketPolicy",
        "s3:PutBucketVersioning",
        "s3:PutBucketObjectLockConfiguration",
        "s3:PutLifecycleConfiguration",
        "s3:BypassGovernanceRetention"
      ],
      "Resource": ["arn:aws:s3:::acme-backups-vault", "arn:aws:s3:::acme-backups-vault/*"],
      "Condition": {
        "ArnNotLike": { "aws:PrincipalArn": "arn:aws:iam::*:role/backup-breakglass" }
      }
    }
  ]
}

A true air gap (offline tape or disks) is the strongest isolation but makes restores slow; immutable storage in a separately administered account is the practical cloud equivalent.

#Encryption and key management

Backups hold everything an attacker wants, so encrypt them, client-side (pgBackRest, WAL-G and restic encrypt before upload) or with KMS. Then guard the keys as carefully as the data, because a backup you can't decrypt is not a backup.

• Keep keys in the backup's security domain. If vault backups use a KMS key in the production account, a production attacker can schedule its deletion (7 to 30 days' wait) and every backup becomes unreadable. Use a key owned by the vault account; give the production writer only kms:GenerateDataKey and the restore role kms:Decrypt.
• Alert on key attacks: ScheduleKeyDeletion, DisableKey and key policy changes. In 2025, attackers with stolen AWS keys re-encrypted S3 objects using customer-provided (SSE-C) keys only they held; deny SSE-C in bucket policies where you don't use it.
• Plan for regional loss. A KMS key lives in one region, so cross-region copies need a key in the destination region (AWS Backup copy jobs re-encrypt with the destination vault's key) or a multi-Region key.
• Escrow passphrases offline, in the break-glass safe with split custody. The engineer who set up the backup tool shouldn't hold the only copy.
• Prove it on every restore test, which is the only evidence the key still exists and works.

#Restore testing and game days

A backup you haven't restored is a hope. Jobs fail silently, archive commands break after upgrades, retention gets shortened, keys disappear, and runbooks reference servers that no longer exist. Test at three levels:

1. Integrity checks, continuously: job exit status, checksums, tool verification (pgbackrest verify, restic check --read-data-subset=5%). Alert when a job fails or doesn't run.
2. Automated restore tests, daily or weekly: restore into a scratch environment, check, time it, tear it down.
3. Game days, a few times a year: a full disaster scenario practised by the people on call.

This automated test for an RDS database measures what matters, the real restore time and the real data loss:

#!/usr/bin/env bash
# restore-test.sh: restore orders to its latest restorable time, check it, delete it.
set -euo pipefail
target="orders-restore-test-$(date -u +%Y%m%d)"
start=$(date +%s)
trap 'aws rds delete-db-instance --db-instance-identifier "$target" --skip-final-snapshot >/dev/null || true' EXIT

aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier orders \
  --target-db-instance-identifier "$target" \
  --use-latest-restorable-time \
  --db-subnet-group-name restore-test \
  --vpc-security-group-ids sg-0a1b2c3d4e5f67890 \
  --no-publicly-accessible --no-multi-az >/dev/null
aws rds wait db-instance-available --db-instance-identifier "$target"

host=$(aws rds describe-db-instances --db-instance-identifier "$target" \
  --query 'DBInstances[0].Endpoint.Address' --output text)
newest=$(psql "host=$host dbname=orders user=restore_check sslmode=require" -Atc \
  "SELECT extract(epoch FROM max(created_at))::bigint FROM orders")

echo "restore took $(( ($(date +%s) - start) / 60 )) min"
echo "newest order was $(( start - newest )) s before the test started"
(( start - newest < 900 )) || { echo "FAIL: data older than 15 minutes"; exit 1; }

The trap deletes the instance however the script exits; the read-only restore_check user's password comes from PGPASSWORD or ~/.pgpass, loaded from a secrets manager. Run it on a schedule and alert if no test has succeeded in eight days. Restore into an isolated network: after ransomware, backups may contain the attacker's tooling, so scan restored systems before reconnecting them.

A game day tests what scripts can't: people, access and documentation. Pick a scenario ("the production account is lost", "the region is down", "someone dropped the payments schema"), give the on-call engineer only the runbook and break-glass access, and time it. You'll find a step needing a permission nobody has, or a restore twice as slow as estimated. Each finding becomes a tracked action item, as in incident response.

#DR strategies, infrastructure as code and GitOps

Backups protect data. Disaster recovery (DR) is the plan for running the whole service elsewhere when a site, region or account is lost:

Choose per system from its RPO and RTO. Every strategy except the first relies on replication, which copies a bad DELETE or ransomware's encryption as faithfully as good data, so every DR strategy still needs point-in-time, immutable backups.

Infrastructure as code is part of your backup. Recovery means recreating networks, clusters, IAM, DNS and load balancers, and nobody remembers which console settings were clicked three years ago. If Terraform describes everything, rebuilding in a new region or account is terraform apply with different variables, provided you've prepared:

• No click-ops drift: anything created by hand can't be rebuilt. Catch drift with scheduled terraform plan.
• Recovery code that doesn't depend on what it recovers: Terraform state in a versioned bucket in another account, and Git repositories mirrored outside your primary Git host.
• Parameterised regions and accounts, with no hard-coded ARNs or AMI IDs.
• Images and secrets in the recovery site: registries and secrets replicated or restorable from the vault.
• GitOps for workloads: a new cluster pointed at the same repository by Argo CD or Flux converges on its own. Use Velero for what isn't in Git, such as persistent volumes:

velero schedule create orders-daily --schedule="0 2 * * *" \
  --include-namespaces orders --ttl 720h0m0s
velero backup get
velero restore create --from-backup orders-daily-20261008020012 --include-namespaces orders

• A known order: identity and break-glass access, then networking, data stores, the platform, applications, and finally DNS cut-over. Rebuild in that order on game days.

#Practice

Acme's orders service runs PostgreSQL 17 on a self-managed EC2 instance with 600 GB of data. A cron job runs pg_dump at 03:00 into an S3 bucket in the production account, versioning off. The business requires RPO 5 minutes and RTO 2 hours, and security requires backups to survive an attacker with production administrator access.

1. List every way the current setup fails.
2. Design a replacement, including the immutability configuration.
3. Restores run at a measured 300 MB/s and WAL replays at about 25 minutes per 10 hours of WAL. With daily base backups, is the RTO met? Show the budget.
4. Describe how you'll prove it keeps working.

aws s3api create-bucket --bucket acme-backups-vault --region eu-west-1 \
  --create-bucket-configuration LocationConstraint=eu-west-1 \
  --object-lock-enabled-for-bucket
aws s3api put-object-lock-configuration --bucket acme-backups-vault \
  --object-lock-configuration '{"ObjectLockEnabled": "Enabled", "Rule": {"DefaultRetention": {"Mode": "COMPLIANCE", "Days": 35}}}'

#Key takeaways

• RPO is how much data you can lose and RTO is how long you can be down; set them per system and check designs against them with measured numbers before a disaster.
• Snapshots, logical dumps and physical backups solve different problems; base backups plus archived WAL give point-in-time recovery to any second.
• Recover from a bad DELETE by restoring a separate copy to just before it and copying rows back, not by rolling production back over later good writes.
• Follow 3-2-1-1-0 and assume attackers hold production admin: keep compliance-mode, Object Lock copies in a separate account that production can only write to.
• Keep backup keys in the backup's own security domain and escrow passphrases offline; replication copies mistakes, so it never replaces backups.
• Test restores automatically and run game days; infrastructure as code, mirrored Git and GitOps make rebuilding a whole environment repeatable.