Production ML
Advanced 15 min read Production ML

Reliability & Incident Response

When the model misbehaves at 2am: what do you do?

Production ML incidents mix classical SRE failures with model-specific ones: silent degradation, drift, data pipeline breaks and decision flips. Response discipline, runbooks, severity modeling and blameless retrospectives keep a model outage from becoming a business incident.

SREincidentrunbookseverityblameless retrodegradationrollbackon-call
Prerequisites: Monitoring Models Production ML Architecture
Before we start

What you will learn

  • Categorize ML incidents across infrastructure, data and model layers
  • Write runbooks that end in rollback, rules-fallback or graceful degradation
  • Run blameless retrospectives that fix process, not people
Introduction

What is this?

Reliability for ML systems is the practice of keeping prediction quality and availability within SLOs despite failures — infrastructure (endpoints down), data (pipeline breaks, staleness), and model behavior (drift, silent decay, decision flips). It pairs classical SRE (SLOs, on-call, runbooks, paging, postmortems) with model-specific response: knowing when to roll back, when to fall back to rules, and how to prove the model — not the platform — is the problem.

Motivation

Why do you need it?

  • ML incidents are noisier than software ones: a 100% available API can be silently giving wrong answers.
  • Response time to the FIRST wrong decision is the reliability currency — dashboards aren't enough, runbooks are.
  • Severity must ladder from 'model drifted' to 'business revenue affected' with communication.
  • Blameless retrospectives convert incidents into engineering improvements that compound.
Think about it

Mental model

Imagine

A hospital emergency response

Think of

ML incident response

Triaging is by severity not by which doctor is available, every patient follows a protocol (runbook), the shift documents what was done (postmortem), and the protocol is revised without blaming the resident (blameless retro). A '2am page, no plan' ML on-call is an ER without triage.

Mechanics

How it works

  • Define SLOs and error budgets per model lane: availability, latency p95, and a quality-proxy SLO.
  • Severity ladder: sev-1 (users harmed / revenue) → sev-2 (degradation) → sev-3 (drift/noise), each with communication paths.
  • Runbooks per alert: symptom → check → most-likely cause → action (rollback, rules fallback, data fix) → verify.
  • On-call rotation with ownership per model, plus a war-room pattern for sev-1s.
  • Blameless retro: timeline, contributing factors, actions with owners and dates; follow up on the actions.

Incident response flow for a model.

Page
Triage
Mitigate
Verify
Investigate
Retro
Real world

In production

  • A payment-model sev-2 pages with a runbook: compare score distributions, diff version, roll back to registry version N, verify chargeback rate within 24h.
  • A search team uses an error budget: when the quality-proxy SLO is exhausted, new feature work pauses until budget recovers.
  • A monthly model review acts as a mini-retro, closing drift and runbook gaps before they become incidents.
Guidelines

Best practices

  • Write the runbook before the model ships, and drill it in staging at least once.
  • Time every SLO in the model's own currency (quality proxy), not only uptime.
  • Automate the first respond: rollback is often a registry pointer flip — make it safe to script.
  • Keep a model-severity matrix so 'drifted' carries the right urgency, and involve the model owner early.
  • Retrospectives fix the system, never blame the operator — feedback loops are noisy by design.
Watch out

Common mistakes

  • On-call for infra only; a model-quality page goes to no one until Monday.
  • Every alert = every response, no severity, so sev-1s compete with field noise.
  • Rollbacks that rerun the exact same defective flow ('we reverted the pointer, it broke again').
  • Postmortems that end at 'root cause found' but never ship the preventive action.
Root cause analysis

Common issues

Error
Quality proxy down but API healthy
Root cause

Model behavior degraded while infrastructure metrics are green

Fix

Add a quality-proxy SLO with its own severity, owner and runbook — infra uptime is not model health.

Error
Rollback flipped, incident persists
Root cause

Rollback to a version that shares the same stale data/bug path

Fix

Treat data fixes and feature-path issues as first-class rollback targets, not only model version.

Error
Same incident recurs weekly
Root cause

Retro produced no accountable preventive action

Fix

Track retro actions as tickets with owners and dates; close the loop at the next review.

When things break

Debugging tips

  • For a model incident, isolate first: is it the data, the model serving the wrong version, or genuine drift?
  • Pull the logged feature snapshot for the affected window before changing anything — you'll need it in the retro.
  • Time-stamp every action; the timeline is the retro's backbone and debugging lens.
  • After mitigation, verify the proxy AND at least one label-backed metric before closing the incident.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The API is green but the model's quality proxy is falling. This incident is:

2

A blameless retrospective's goal is to find whose fault the incident was.

Go deeper

Official documentation

Google SRE — Error Budgets
docs
Google SRE — Postmortem culture
docs