# Monitoring & incident response > Log the security events that matter, centralise them, design alerts people act on, and run an incident from detection to blameless postmortem. DevSecOps Engineer roadmap · Operate & recover · 20 min read · https://levelcode.org/roadmaps/devsecops/incident-response Every control in this roadmap reduces the chance of a breach; none reduces it to zero. What separates a minor incident from a disaster is usually time: how quickly you notice, and how effectively you respond. This lesson covers which events to log, how to centralise them, how to write alerts people act on, the standard incident response lifecycle, a leaked-credential incident minute by minute, and the blameless postmortem that makes the next one less likely. ## What to log Debug logs rarely answer an investigation's questions: *who* did *what*, *to which* resource, *from where*, *when*, and *did it succeed*? Security logging means deliberately emitting events for actions that matter. | Category | Examples | |---|---| | Authentication | Logins (success and failure), logouts, MFA enrolment and removal, password resets, new API tokens | | Authorisation | Access denied responses, especially repeated ones; attempts to reach other tenants' data | | Privilege changes | Role grants, admin promotions, new users, group membership changes | | Sensitive data access | Bulk exports, report downloads, reading secrets from a vault | | Configuration changes | Security settings, feature flags controlling security, firewall and IAM changes | | Security control health | Logging disabled, agent stopped, alert rule deleted (attackers do this early) | Write these as **structured events** with consistent fields, so they can be queried across services: ```json { "timestamp": "2026-10-06T14:02:11.482Z", "event": "auth.login", "outcome": "failure", "reason": "invalid_password", "user": { "id": "u_8f2c", "name": "jane@acme.example" }, "source": { "ip": "203.0.113.45", "user_agent": "python-requests/2.32.3" }, "service": "orders-api", "environment": "production", "request_id": "req_01J9Z3K7Q2", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736" } ``` Use UTC timestamps and NTP-synced clocks, or you can't order events across systems. The `request_id` and `trace_id` link a security event to everything else that request did. Never log passwords (even wrong ones, often a typo of the right one), session tokens, API keys, card numbers, or personal data you don't need. Logs are copied widely and kept for years; a secret in a log is a leaked secret. Turn on platform audit logs too: cloud control-plane logs (an organisation-wide AWS CloudTrail trail, Google Cloud Audit Logs, Azure Activity Log), Kubernetes audit logs, identity provider sign-ins, and source control and CI audit logs. ## Centralised logging and SIEM Logs scattered across hosts are useless in an incident: the attacker may have deleted them, and nobody can correlate five systems by hand at 3 a.m. A typical central pipeline: 1. **Collect.** An agent or collector (Fluent Bit, Vector, the OpenTelemetry Collector, or the cloud's native log routing) ships logs off the host as they're written. 2. **Normalise.** Map fields to a common schema (such as Elastic Common Schema or OCSF), so `src_ip`, `clientIp` and `sourceIPAddress` all become `source.ip`. 3. **Store.** Hot, searchable storage for recent weeks; cheap object storage for long-term retention. Many regulations require a year or more. 4. **Detect.** Rules run continuously against incoming events and raise alerts. 5. **Respond.** Alerts become cases with an owner, a status and an audit trail. A **SIEM** (security information and event management) system such as Splunk, Microsoft Sentinel or Elastic Security bundles storage, search, correlation and detection. The concepts transfer between them. Protect the logs themselves: ship them to a **separate account** that production administrators can't modify, enable immutability (such as S3 Object Lock), and alert when a source goes quiet. An attacker with admin rights often calls `StopLogging` first. ## Designing alerts people act on An alert is a request for a human's attention. Send too many false ones and people stop reading them; that **alert fatigue** is how real incidents get missed. Good alerts are: - **Actionable**: there is something specific the responder should do. "CPU is high on a security tool" is not. - **High signal**: most firings are real or at least worth checking. Measure it, and tune or delete rules that are always benign. - **Contextual**: who, what, where, and links to the relevant logs, not just a rule name. - **Linked to a runbook**: the first five steps, written in advance, when nobody is stressed. - **Routed by severity**: page a human only for things that can't wait until morning. | Alert | Severity | Route | |---|---|---| | Root or break-glass account sign-in | Critical | Page on-call security immediately | | Cloud provider reports an exposed access key | Critical | Page | | CloudTrail logging stopped or trail deleted | High | Page | | New IAM user or access key created outside Terraform | High | Page in business hours, ticket otherwise | | 50 failed logins for one account in 5 minutes | Medium | Ticket, and auto-lock the account | | New dependency vulnerability in a running image | Low | Ticket to the owning team | Write detections **as code**, reviewed and tested. **Sigma** is a vendor-neutral YAML detection format that converts into most SIEMs' query languages: ```yaml title: IAM user or access key created outside Terraform id: ffaeba52-7297-4e2a-bd60-9bc1279e6538 status: stable description: >- Identity changes should only come from the terraform-apply role. Anything else may be an attacker establishing persistence with a stolen credential. logsource: product: aws service: cloudtrail detection: selection: eventSource: iam.amazonaws.com eventName: - CreateUser - CreateAccessKey - CreateLoginProfile - AttachUserPolicy filter_terraform: userIdentity.arn|contains: ':assumed-role/terraform-apply/' condition: selection and not filter_terraform falsepositives: - Break-glass use during an incident (separately alerted) level: high ``` > [!TIP] > Test detections by triggering them. A rule that has never fired may simply be broken: a renamed field or a missing log source fails silently. Run harmless simulations on a schedule, such as creating and deleting a test user from a sandbox role, and check that the alert arrives. ## The incident response lifecycle The model used across the industry comes from NIST's incident handling guide, SP 800-61. Its latest revision (Rev. 3, 2025) maps incident response onto the NIST Cybersecurity Framework 2.0 functions, and replaces the classic four-phase lifecycle of the previous revision (Rev. 2). That lifecycle is still the practical working model most runbooks use: 1. **Preparation.** Logging and alerting, runbooks, an on-call rota, a contact list (legal, communications, cloud provider support), responder access, tested backups, and tabletop exercises. 2. **Detection and analysis.** An alert fires or someone reports something. Is it real? What is affected? How bad is it? Declare an incident as soon as it's plausibly real: declaring early and downgrading costs little; declaring late costs a lot. 3. **Containment, eradication and recovery.** *Contain* first: stop the damage spreading (revoke credentials, isolate hosts, block IPs) while **preserving evidence** such as disk snapshots and logs. *Eradicate*: remove the attacker's access and footholds, and fix the weakness they used. *Recover*: restore normal service, carefully, with extra monitoring. 4. **Post-incident activity.** A postmortem, action items, and updates to detection and runbooks. This feeds back into preparation. Define roles, or an incident becomes ten people running commands at once: - **Incident commander (IC)**: coordinates, decides, and keeps the big picture. The IC doesn't type commands. - **Operations lead(s)**: investigate and make changes, announcing each action before taking it. - **Communications lead**: updates stakeholders, customers and, where required, regulators on a fixed schedule. - **Scribe**: records a timestamped log of findings and actions. This becomes the postmortem timeline. ## Worked example: a leaked access key Here is a realistic incident, condensed. Times are UTC on a single day. | Time | Event | |---|---| | 13:58 | While debugging a CI failure, a developer commits a `.env` file containing the access key of IAM user `ci-legacy` to a public repository. | | 14:01 | First API call with the key from `203.0.113.45`: `sts:GetCallerIdentity`. Automated scanners watch public commits and test keys within minutes. | | 14:02 | `s3:ListBuckets` succeeds; `s3:GetObject` succeeds on three objects in the build-artifacts bucket. | | 14:03 | AWS, notified through GitHub's secret scanning partner programme, attaches a quarantine policy to the user and opens a support case. | | 14:04 | `ec2:RunInstances` in two Regions and `iam:CreateUser` (`backup-admin`) fail with `AccessDenied`. | | 14:05 | Alerts fire: the "exposed key" support case and the "IAM user created outside Terraform" Sigma rule. On-call is paged. | | 14:09 | On-call acknowledges, declares a SEV-2 incident, and takes the IC role; a second engineer becomes operations lead. | | 14:12 | Containment begins. | **Contain.** The quarantine policy blocks only some actions. The operations lead attaches an explicit deny-all to the user, then deactivates the key. The order matters: temporary credentials the attacker may already have minted with `sts:GetSessionToken` survive key deactivation, but they are still evaluated against the user's policies, so the deny blocks them too. ```json { "Version": "2012-10-17", "Statement": [{ "Effect": "Deny", "Action": "*", "Resource": "*" }] } ``` ```bash aws iam put-user-policy --user-name ci-legacy \ --policy-name incident-2026-031-deny-all --policy-document file://deny-all.json aws iam update-access-key --user-name ci-legacy \ --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive ``` The key is **deactivated, not deleted**: its ID ties the attacker's activity together in the logs. **Analyse scope.** CloudTrail's event history is per Region, so search them all, by the key and by the user (in case the attacker created a second key): ```bash for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do aws cloudtrail lookup-events --region "$r" --start-time 2026-10-06T13:00:00Z \ --lookup-attributes AttributeKey=Username,AttributeValue=ci-legacy \ --query 'Events[].[EventTime,EventName]' --output text | sed "s/^/$r\t/" done ``` ```text us-east-1 2026-10-06T14:04:51+00:00 CreateUser us-east-1 2026-10-06T14:01:37+00:00 GetCallerIdentity eu-west-1 2026-10-06T14:04:22+00:00 RunInstances eu-west-1 2026-10-06T14:02:05+00:00 ListBuckets ap-southeast-1 2026-10-06T14:04:30+00:00 RunInstances ``` Event history covers management events only. The `GetObject` calls are **data events**, found only in the organisation trail's S3 logs in the SIEM. Those show three build artifacts read, no customer data. The analysis confirms: no instances launched, no user created (`aws iam list-users` and the CloudTrail `errorCode` fields agree), no other keys used. **Eradicate.** Delete the key. Then remove the reason it existed: the CI job that used `ci-legacy` is moved to an OIDC role (as in the cloud IAM lesson) and the user is deleted. Removing the commit from Git history is good hygiene, but it changes nothing for security: the key was public, so it is burned regardless. **Recover.** CI deploys with the new role, and the team watches for activity from the attacker's IP range for a week. The three exposed artifacts are reviewed for embedded secrets; one contains a test API token, which is rotated. **Communicate.** Internal updates go out every 30 minutes. Legal confirms no customer data was accessed, so no external notification is required; the conclusion and its evidence are recorded. > [!IMPORTANT] > Preserve evidence before you clean up. Export the relevant CloudTrail events, snapshot affected disks, and keep the deactivated key's metadata until analysis is done. If you delete first, you may never learn what the attacker did, and you can't prove what they didn't. ## Blameless postmortems Within a few days, the team writes a **postmortem**, for learning rather than judgement. A **blameless** postmortem assumes people made reasonable decisions with the information and tools they had; if one human slip could cause an incident, the system was fragile and *that* is what to fix. Blame makes people hide mistakes, destroying exactly the information you need. Compare: - Blameful: "Dev X carelessly committed the key." - Blameless: "A long-lived key existed for a CI job; `.env` wasn't in `.gitignore`; push protection wasn't enabled on the repository, so nothing stopped the commit." A useful structure: ```text Title: Exposed AWS access key for ci-legacy (SEV-2), 2026-10-06 Summary: What happened and the impact, in three sentences. Impact: Three build artifacts read by an unknown party; no customer data; no downtime. Timeline: From the scribe's log, in UTC. Contributing factors: - A long-lived IAM user key was still used by one CI job after the OIDC migration. - The repository template lacked .env in .gitignore. - Secret scanning push protection was not enabled for the repository. - The key's policy allowed s3:GetObject on all artifacts, not one prefix. What went well: Provider quarantine plus our detection paged within 4 minutes. What was hard: No runbook for "exposed key"; responders searched Regions one by one. Action items (owner, due date, ticket): 1. Remove all remaining IAM users with access keys from CI. platform, 2026-10-20 2. Enable push protection organisation-wide. security, 2026-10-09 3. Add .env to the repository template's .gitignore. devex, 2026-10-09 4. Write the exposed-credential runbook, with the scripts above. security, 2026-10-16 ``` Every action item has an owner and a date and is tracked like any other work; a postmortem whose actions never happen is a story, not an improvement. Share postmortems widely: other teams have the same contributing factors. ## Practice Your SIEM shows these CloudTrail events, all from IP `198.51.100.23`, made with access key `AKIAI44QH8DHBEXAMPLE` belonging to IAM user `reporting-bot`, which normally only calls `athena:StartQueryExecution` from your office: ```text 09:12:03 sts:GetCallerIdentity success 09:12:40 iam:ListAttachedUserPolicies success 09:13:15 iam:CreateAccessKey (userName=reporting-bot) success -> AKIAJQ5CMLBEXAMPLE2 09:14:02 s3:ListBuckets success 09:15:30 cloudtrail:StopLogging AccessDenied ``` 1. Write a short analysis: what did the attacker try to do, and which event worries you most? 2. List your containment steps in order, with the commands. 3. Write one Sigma detection that would have caught this earlier. **Solution** **Analysis.** The attacker validated the key, enumerated its permissions, then established **persistence** with a second access key so that rotating the original wouldn't lock them out. They explored storage and tried to disable logging. The most serious event is `CreateAccessKey`: there is now a second credential the attacker controls. `StopLogging` being denied is the SCP guardrail working. **Containment, in order:** ```bash # 1. Deny everything for the user: covers both keys and any session tokens. aws iam put-user-policy --user-name reporting-bot \ --policy-name incident-deny-all --policy-document file://deny-all.json # 2. Deactivate both keys (keep them for the investigation). aws iam update-access-key --user-name reporting-bot --access-key-id AKIAI44QH8DHBEXAMPLE --status Inactive aws iam update-access-key --user-name reporting-bot --access-key-id AKIAJQ5CMLBEXAMPLE2 --status Inactive # 3. Confirm there are no other keys, a console password, or new users. aws iam list-access-keys --user-name reporting-bot aws iam get-login-profile --user-name reporting-bot # NoSuchEntity is the good answer ``` Then search every Region and the data-event logs by `Username`, export the evidence, and only then delete the keys. Replace the bot's key with a role without IAM write permissions. **Detection.** An IAM user creating an access key is suspicious once people sign in through SSO; so is a new key that isn't created by Terraform: ```yaml title: IAM user created an access key id: 0b6f8a43-58c1-4c8e-9a51-2f6d0f6d7c11 status: experimental logsource: product: aws service: cloudtrail detection: selection: eventSource: iam.amazonaws.com eventName: CreateAccessKey userIdentity.type: IAMUser condition: selection falsepositives: - Manual key rotation by a human user (should be rare once users are replaced by SSO) level: high ``` The lesson's rule (IAM changes outside `terraform-apply`) would also have fired at 09:13, and a behavioural rule, "`reporting-bot` used from outside the office range", at 09:12. ## Key takeaways - Log security events as structured records answering who, what, which resource, from where, when and with what outcome, and never log secrets. - Centralise logs in a separate, tamper-resistant account, normalise their fields, and alert when a source goes quiet. - Good alerts are actionable, high-signal, contextual, linked to a runbook and routed by severity; write and test detections as code, for example in Sigma. - Follow the lifecycle: prepare, detect and analyse, contain/eradicate/recover, then learn; assign an incident commander, a scribe and a communications lead. - For a leaked credential: block it immediately (deny policy, deactivate), search all Regions and data events, look for persistence, preserve evidence, then remove the long-lived credential entirely. - Run blameless postmortems that fix systems, with owned, dated action items. ## Quiz 1. During a leaked-key incident, why attach an explicit deny-all policy to the IAM user in addition to deactivating the access key? A. Deactivated keys can be reactivated by anyone B. Temporary credentials already minted from the key with GetSessionToken survive deactivation, but are still blocked by the user's deny policy C. Deactivation takes 24 hours to propagate D. CloudTrail does not log deactivated keys **Answer: B.** Session tokens derived from the key remain valid after the key is disabled. They are evaluated against the user's current policies, so an explicit deny cuts them off too. 2. Which of these should never appear in an authentication log event? A. The source IP address B. The outcome (success or failure) C. The password that was entered D. The request ID **Answer: C.** Logs are widely copied and retained. Even failed passwords are often a near-miss of the real one, so logging them leaks credentials. 3. A detection rule has fired 300 times this month and every instance was benign. What is the best action? A. Tune or rewrite it so it fires only on suspicious cases, or delete it if it can't be made useful B. Leave it as is, because more alerts means more security C. Raise its severity so people pay attention D. Route it to a shared mailbox nobody reads **Answer: A.** Noisy alerts cause fatigue, and fatigue causes real alerts to be missed. Every alert should be worth a human's attention. 4. Which statement best reflects a blameless postmortem? A. The engineer who made the mistake writes an apology B. Postmortems are only written for SEV-1 incidents C. The goal is to find the person responsible so it doesn't happen again D. If one human slip could cause the incident, the system lacked safeguards, and the action items should add them **Answer: D.** Blamelessness keeps people honest about what happened. Fixes target the contributing factors, such as missing push protection or long-lived keys, rather than individuals.