AI Security
Advanced 13 min read AI Security

AI Incident Response

The playbook for when it goes wrong

When an AI system misbehaves — leaks data, takes an odd action, or gets jailbroken at scale — you need a runbook: detect via logs, contain by killing tools and traffic, triage the blast radius, and root-cause with model and trace data.

incidentresponserunbookcontainmentblast radiusforensics
Prerequisites: Secure Deployment
Before we start

What you will learn

  • Detect AI incidents from telemetry
  • Contain via tool/traffic cutoffs
  • Root-cause from traces and model data
Introduction

What is this?

AI incident response is a runbook for model-system failures: detection (anomaly metrics, sentries, user reports), containment (kill write tools, freeze traffic, rotate keys), triage (blast radius — which tenants/data/tools touched), and root cause (traces, model versions, prompts). Rehearsed, not improvised.

Motivation

Why do you need it?

  • AI failures can cascade quietly through tool chains.
  • Speed of containment decides damage (leaks, writes).
  • Trace-rich systems make root cause actually possible.
  • A rehearsed runbook beats heroics under stress.
Think about it

Mental model

Imagine

A code red at an aquarium

Think of

Incident response

When a tank leaks, you don't argue about the fish — you isolate the tank (containment), check which tanks share water (blast radius), then train the pump (root cause). Drills (rehearsals) make the whole building move together. Same for an agent that's done something unplanned.

Mechanics

How it works

  • Detect: anomaly dashboards, sentry flags, abuse reports.
  • Classify: is it data, action, or jailbreak class?
  • Contain: disable write tools, cut egress, freeze the rare path.
  • Triage: who/what was touched; preserve traces.
  • Root-cause: replay traces; fix; add eval case; postmortem.

The runbook.

Detect
Classify
Contain
Triage
Fix
Real world

In production

  • Runbooks with on-call rotation for AI systems.
  • Kill-switch toggles for write tools in dashboards.
  • Telemetry that captures full traces for every incident.
Guidelines

Best practices

  • Kill switches: instantly disable write-heavy tools.
  • Capture traces BEFORE fixing (evidence first).
  • Postmortems add the failure as an eval/golden case.
  • Practice with red-team drills.
Watch out

Common mistakes

  • No telemetry = guessing at blast radius.
  • Fixing code before capturing traces (evidence lost).
  • Containment rested on disabling a tool that was still warm in caches.
Root cause analysis

Common issues

Error
Containment didn't stop writes
Root cause

Cache/queue still executing deferred tools

Fix

Drain or purge queues; kill at the tool boundary too.

Error
Can't reproduce
Root cause

Prompts/models unpinned

Fix

Pin model+prompt versions in the trace.

When things break

Debugging tips

  • Silent incidents → rely on sentries + sample audits; they find what alerts miss.
  • Blast radius unknown → trace join across tenants and tools.
  • Repeat incidents → root causes not converted to eval cases.
Interviews

Interview checkpoint

Practice

Hands-on lab

Draft an Incident Runbook Prototype
35 min · advanced
In progress
Objectives
  • Detect an anomaly from a synthetic log
  • Contain via a kill switch
  • Preserve traces for root cause

Draft an Incident Runbook Prototype

35 min · advanced

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

First priority during an AI incident:

2

Traces should be captured before fixes.

Go deeper

Official documentation

Incident response (Google)
docs