Cloud Security
Advanced 15 min read Cloud Security

Incident Response & Playbooks

The playbook you hope never to use properly

IR is a game plan for being attacked, not a wish: detection → containment → eradication → recovery → lessons. Cloud IR leans on native speed: isolate resources, snapshot evidence, revoke keys, restore from immutable snapshots, and quarantine the attack path.

incident responsecontainmentquarantineforensicsplaybookTOCTOU
Prerequisites: Logging & Audit
Before we start

What you will learn

  • Structure an IR loop (detect→contain→eradicate→recover→learn)
  • Use cloud-native containment (SG quarantine, snapshot, key revoke)
  • Draft and rehearse playbooks per severity class
In 60 seconds

A good IR loop keeps scope small and impact contained while later phases rebuild. In the cloud that translates to concrete knobs: deny traffic via NACL/SG, detach credentials, snapshot the moment, restore from versioned backup, and capture CloudTrail before touching anything.

Introduction

What is this?

Phases: detect (Alerting/SIEM), triage (severity, blast radius), contain (killner traffic/network/keys), eradicate (remove the mechanism), recover (restore capacity/data), lessons (retro, controls). Cloud levers: stop instances, revoke IAM/keys (rotate before disable), detach/disable access keys, SG/NACL quarantine, snapshot EBS/volumes/TFS for forensics, object-lock the audit bucket, restore from immutable backup, change passwords/rotate all creds. Evidence: CloudTrail timestamps, instance/volume snapshot, memory/RAM-disk images, DNS/log forensics. Posture: prep (playbooks, runbooks, RACI, war-room channel), drills quarterly, IR tooling (SSM forensics scripts, incident-env workflows).

Motivation

Why do you need it?

  • Incidents are events with timers; playbooks remove the 'what do I do' paralysis.
  • Cloud containment levers act in seconds, but only if you know which one to pull.
  • Snapshotting before reclaiming preserves the story for root-cause.
  • A rehearsed drill is the only way playbooks behave as expected at 3am.
Think about it

Mental model

Imagine

A fire drill: alarm (detect), isolate the floor (contain), remove the cause (eradicate), rebuild (recover), and chart the drill (lessons)

Think of

Incident response lifecycle

You don't wait for the inspector to teach you the exit route — you rehearse it, and the post-mortem improves the next drill.

Mechanics

How it works

  • Detection triggers from GuardDuty/Defender/audit-alerts → IR channel.
  • Triage confirms real attack + scope; the war room opens; RACI invoked.
  • Containment: quarantine in minutes (network isolation, key revoke) before scale-out.
  • Eradication: remove the access/implants; snapshot evidence first.
  • Recovery: restore from trusted backups + build-time; audit keeps forensic chain; retro updates controls.

IR loop.

Detect
Contain
Eradicate
Recover
Learn
Hands-on

Examples

Containment commands at the border

Containment commands at the border
# 1) quarantine network (IAM-widest deny)
aws ec2 create-network-acl ...  # deny all egress/ingress on suspicious subnets
# 2) snapshot evidence before mutation
aws ec2 create-snapshot --volume-id <vol-id> --description "IR-SNAPSHOT-<ts>:preserve"
# 3) rotate keys (not just delete): create new, migrate, delete old
# 4) object-lock audit trail:
aws s3api put-object-legal-hold --bucket sec-logs --key trail.gz --legal-hold {'Status':'ON'}

Seconds to isolate, evidence preserved, credentials rotated, audit sacrosanct — the IR first-15-minutes checklist.

Real world

In production

  • War-room channel + playbook per severity; on-call triggered from detection findings.
  • Forensic vault: spot-instance snapshots + audit lock for retrospective reconstruction.
  • Quarterly tabletop with fake cloud incident: team walks detection→contain→recover without breaking prod.
Guidelines

Best practices

  • Rehearse playbooks; a playbook nobody has run is a checklist someone will improvise at 3am.
  • Snapshot twice: one permissive-forensic, one operational-restore — separate concerns.
  • Rotate everything credential when containment begins (compromise scope unknown).
  • Capture CloudTrail + key logs before killing the instance (evidence dies with the box).
  • Blameless retro: controls, not culprits, are the output.
Watch out

Common mistakes

  • Ordering 'delete and ignore' — that's how the same hole reopens next quarter.
  • Containing by killing the instance before evidence capture.
  • No rehearsal: playbooks exist in a doc nobody opened.
  • Fixing the symptom (IP blocked) while the credential/footprint persists.
When things break

Debugging tips

  • Alert that's actually benign → annotate and hit the 'why' before silencing (recurrence = new attack path).
  • Containment rule didn't apply → test the NACL/SG route + source/dest checks at drill time.
  • Restore spike fails → backup immutability/config mismatch; test a cold restore quarterly.
  • Forensic chain gaps → correlate CloudTrail window with the incident env timeline.
Interviews

Interview checkpoint

Practice

Hands-on lab

Tabletop the First 15 Minutes
45 min · advanced
In progress
Objectives
  • Define a war-room channel + RACI
  • Rehearse contain-recover commands against a fake compromise
  • Produce the forensic + containment checklist

Tabletop the First 15 Minutes

45 min · advanced

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Before killing a compromised instance you must:

2

The output of a good retro is:

Go deeper

Official documentation

AWS IR whitepaper
docs
NIST SP 800-61
spec
Azure IR guidance
docs