Monitoring & incident response

Every control in this roadmap reduces the chance of a breach; none reduces it to zero. What separates a minor incident from a disaster is usually time: how quickly you notice, and how effectively you respond. This lesson covers which events to log, how to centralise them, how to write alerts people act on, the standard incident response lifecycle, a leaked-credential incident minute by minute, and the blameless postmortem that makes the next one less likely.

#What to log

Debug logs rarely answer an investigation's questions: who did what, to which resource, from where, when, and did it succeed? Security logging means deliberately emitting events for actions that matter.

Write these as structured events with consistent fields, so they can be queried across services:

{
  "timestamp": "2026-10-06T14:02:11.482Z",
  "event": "auth.login",
  "outcome": "failure",
  "reason": "invalid_password",
  "user": { "id": "u_8f2c", "name": "[email protected]" },
  "source": { "ip": "203.0.113.45", "user_agent": "python-requests/2.32.3" },
  "service": "orders-api",
  "environment": "production",
  "request_id": "req_01J9Z3K7Q2",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}

Use UTC timestamps and NTP-synced clocks, or you can't order events across systems. The request_id and trace_id link a security event to everything else that request did.

Never log passwords (even wrong ones, often a typo of the right one), session tokens, API keys, card numbers, or personal data you don't need. Logs are copied widely and kept for years; a secret in a log is a leaked secret.

Turn on platform audit logs too: cloud control-plane logs (an organisation-wide AWS CloudTrail trail, Google Cloud Audit Logs, Azure Activity Log), Kubernetes audit logs, identity provider sign-ins, and source control and CI audit logs.

#Centralised logging and SIEM

Logs scattered across hosts are useless in an incident: the attacker may have deleted them, and nobody can correlate five systems by hand at 3 a.m. A typical central pipeline:

1. Collect. An agent or collector (Fluent Bit, Vector, the OpenTelemetry Collector, or the cloud's native log routing) ships logs off the host as they're written.
2. Normalise. Map fields to a common schema (such as Elastic Common Schema or OCSF), so src_ip, clientIp and sourceIPAddress all become source.ip.
3. Store. Hot, searchable storage for recent weeks; cheap object storage for long-term retention. Many regulations require a year or more.
4. Detect. Rules run continuously against incoming events and raise alerts.
5. Respond. Alerts become cases with an owner, a status and an audit trail.

A SIEM (security information and event management) system such as Splunk, Microsoft Sentinel or Elastic Security bundles storage, search, correlation and detection. The concepts transfer between them.

Protect the logs themselves: ship them to a separate account that production administrators can't modify, enable immutability (such as S3 Object Lock), and alert when a source goes quiet. An attacker with admin rights often calls StopLogging first.

#Designing alerts people act on

An alert is a request for a human's attention. Send too many false ones and people stop reading them; that alert fatigue is how real incidents get missed. Good alerts are:

• Actionable: there is something specific the responder should do. "CPU is high on a security tool" is not.
• High signal: most firings are real or at least worth checking. Measure it, and tune or delete rules that are always benign.
• Contextual: who, what, where, and links to the relevant logs, not just a rule name.
• Linked to a runbook: the first five steps, written in advance, when nobody is stressed.
• Routed by severity: page a human only for things that can't wait until morning.

Write detections as code, reviewed and tested. Sigma is a vendor-neutral YAML detection format that converts into most SIEMs' query languages:

title: IAM user or access key created outside Terraform
id: ffaeba52-7297-4e2a-bd60-9bc1279e6538
status: stable
description: >-
  Identity changes should only come from the terraform-apply role. Anything else
  may be an attacker establishing persistence with a stolen credential.
logsource:
  product: aws
  service: cloudtrail
detection:
  selection:
    eventSource: iam.amazonaws.com
    eventName:
      - CreateUser
      - CreateAccessKey
      - CreateLoginProfile
      - AttachUserPolicy
  filter_terraform:
    userIdentity.arn|contains: ':assumed-role/terraform-apply/'
  condition: selection and not filter_terraform
falsepositives:
  - Break-glass use during an incident (separately alerted)
level: high

#The incident response lifecycle

The model used across the industry comes from NIST's incident handling guide, SP 800-61. Its latest revision (Rev. 3, 2025) maps incident response onto the NIST Cybersecurity Framework 2.0 functions, and replaces the classic four-phase lifecycle of the previous revision (Rev. 2). That lifecycle is still the practical working model most runbooks use:

1. Preparation. Logging and alerting, runbooks, an on-call rota, a contact list (legal, communications, cloud provider support), responder access, tested backups, and tabletop exercises.
2. Detection and analysis. An alert fires or someone reports something. Is it real? What is affected? How bad is it? Declare an incident as soon as it's plausibly real: declaring early and downgrading costs little; declaring late costs a lot.
3. Containment, eradication and recovery. Contain first: stop the damage spreading (revoke credentials, isolate hosts, block IPs) while preserving evidence such as disk snapshots and logs. Eradicate: remove the attacker's access and footholds, and fix the weakness they used. Recover: restore normal service, carefully, with extra monitoring.
4. Post-incident activity. A postmortem, action items, and updates to detection and runbooks. This feeds back into preparation.

Define roles, or an incident becomes ten people running commands at once:

• Incident commander (IC): coordinates, decides, and keeps the big picture. The IC doesn't type commands.
• Operations lead(s): investigate and make changes, announcing each action before taking it.
• Communications lead: updates stakeholders, customers and, where required, regulators on a fixed schedule.
• Scribe: records a timestamped log of findings and actions. This becomes the postmortem timeline.

#Worked example: a leaked access key

Here is a realistic incident, condensed. Times are UTC on a single day.

Contain. The quarantine policy blocks only some actions. The operations lead attaches an explicit deny-all to the user, then deactivates the key. The order matters: temporary credentials the attacker may already have minted with sts:GetSessionToken survive key deactivation, but they are still evaluated against the user's policies, so the deny blocks them too.

{
  "Version": "2012-10-17",
  "Statement": [{ "Effect": "Deny", "Action": "*", "Resource": "*" }]
}

aws iam put-user-policy --user-name ci-legacy \
  --policy-name incident-2026-031-deny-all --policy-document file://deny-all.json
aws iam update-access-key --user-name ci-legacy \
  --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive

The key is deactivated, not deleted: its ID ties the attacker's activity together in the logs.

Analyse scope. CloudTrail's event history is per Region, so search them all, by the key and by the user (in case the attacker created a second key):

for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
  aws cloudtrail lookup-events --region "$r" --start-time 2026-10-06T13:00:00Z \
    --lookup-attributes AttributeKey=Username,AttributeValue=ci-legacy \
    --query 'Events[].[EventTime,EventName]' --output text | sed "s/^/$r\t/"
done

us-east-1	2026-10-06T14:04:51+00:00	CreateUser
us-east-1	2026-10-06T14:01:37+00:00	GetCallerIdentity
eu-west-1	2026-10-06T14:04:22+00:00	RunInstances
eu-west-1	2026-10-06T14:02:05+00:00	ListBuckets
ap-southeast-1	2026-10-06T14:04:30+00:00	RunInstances

Event history covers management events only. The GetObject calls are data events, found only in the organisation trail's S3 logs in the SIEM. Those show three build artifacts read, no customer data. The analysis confirms: no instances launched, no user created (aws iam list-users and the CloudTrail errorCode fields agree), no other keys used.

Eradicate. Delete the key. Then remove the reason it existed: the CI job that used ci-legacy is moved to an OIDC role (as in the cloud IAM lesson) and the user is deleted. Removing the commit from Git history is good hygiene, but it changes nothing for security: the key was public, so it is burned regardless.

Recover. CI deploys with the new role, and the team watches for activity from the attacker's IP range for a week. The three exposed artifacts are reviewed for embedded secrets; one contains a test API token, which is rotated.

Communicate. Internal updates go out every 30 minutes. Legal confirms no customer data was accessed, so no external notification is required; the conclusion and its evidence are recorded.

#Blameless postmortems

Within a few days, the team writes a postmortem, for learning rather than judgement. A blameless postmortem assumes people made reasonable decisions with the information and tools they had; if one human slip could cause an incident, the system was fragile and that is what to fix. Blame makes people hide mistakes, destroying exactly the information you need.

Compare:

• Blameful: "Dev X carelessly committed the key."
• Blameless: "A long-lived key existed for a CI job; .env wasn't in .gitignore; push protection wasn't enabled on the repository, so nothing stopped the commit."

A useful structure:

Title:        Exposed AWS access key for ci-legacy (SEV-2), 2026-10-06
Summary:      What happened and the impact, in three sentences.
Impact:       Three build artifacts read by an unknown party; no customer data; no downtime.
Timeline:     From the scribe's log, in UTC.
Contributing factors:
  - A long-lived IAM user key was still used by one CI job after the OIDC migration.
  - The repository template lacked .env in .gitignore.
  - Secret scanning push protection was not enabled for the repository.
  - The key's policy allowed s3:GetObject on all artifacts, not one prefix.
What went well:  Provider quarantine plus our detection paged within 4 minutes.
What was hard:   No runbook for "exposed key"; responders searched Regions one by one.
Action items (owner, due date, ticket):
  1. Remove all remaining IAM users with access keys from CI.     platform, 2026-10-20
  2. Enable push protection organisation-wide.                    security, 2026-10-09
  3. Add .env to the repository template's .gitignore.            devex,    2026-10-09
  4. Write the exposed-credential runbook, with the scripts above. security, 2026-10-16

Every action item has an owner and a date and is tracked like any other work; a postmortem whose actions never happen is a story, not an improvement. Share postmortems widely: other teams have the same contributing factors.

#Practice

Your SIEM shows these CloudTrail events, all from IP 198.51.100.23, made with access key AKIAI44QH8DHBEXAMPLE belonging to IAM user reporting-bot, which normally only calls athena:StartQueryExecution from your office:

09:12:03  sts:GetCallerIdentity            success
09:12:40  iam:ListAttachedUserPolicies     success
09:13:15  iam:CreateAccessKey  (userName=reporting-bot)  success  -> AKIAJQ5CMLBEXAMPLE2
09:14:02  s3:ListBuckets                   success
09:15:30  cloudtrail:StopLogging           AccessDenied

1. Write a short analysis: what did the attacker try to do, and which event worries you most?
2. List your containment steps in order, with the commands.
3. Write one Sigma detection that would have caught this earlier.

# 1. Deny everything for the user: covers both keys and any session tokens.
aws iam put-user-policy --user-name reporting-bot \
  --policy-name incident-deny-all --policy-document file://deny-all.json

# 2. Deactivate both keys (keep them for the investigation).
aws iam update-access-key --user-name reporting-bot --access-key-id AKIAI44QH8DHBEXAMPLE --status Inactive
aws iam update-access-key --user-name reporting-bot --access-key-id AKIAJQ5CMLBEXAMPLE2 --status Inactive

# 3. Confirm there are no other keys, a console password, or new users.
aws iam list-access-keys --user-name reporting-bot
aws iam get-login-profile --user-name reporting-bot   # NoSuchEntity is the good answer

title: IAM user created an access key
id: 0b6f8a43-58c1-4c8e-9a51-2f6d0f6d7c11
status: experimental
logsource:
  product: aws
  service: cloudtrail
detection:
  selection:
    eventSource: iam.amazonaws.com
    eventName: CreateAccessKey
    userIdentity.type: IAMUser
  condition: selection
falsepositives:
  - Manual key rotation by a human user (should be rare once users are replaced by SSO)
level: high

#Key takeaways

• Log security events as structured records answering who, what, which resource, from where, when and with what outcome, and never log secrets.
• Centralise logs in a separate, tamper-resistant account, normalise their fields, and alert when a source goes quiet.
• Good alerts are actionable, high-signal, contextual, linked to a runbook and routed by severity; write and test detections as code, for example in Sigma.
• Follow the lifecycle: prepare, detect and analyse, contain/eradicate/recover, then learn; assign an incident commander, a scribe and a communications lead.
• For a leaked credential: block it immediately (deny policy, deactivate), search all Regions and data events, look for persistence, preserve evidence, then remove the long-lived credential entirely.
• Run blameless postmortems that fix systems, with owned, dated action items.