Alerting and SLOs for Models
The numbers that define 'good enough' to be trustworthy
Model SLOs bind monitoring to trust: availability, latency, drift and quality targets that define 'good'. Alerting turns those SLOs into owners with pages on breach — but only when the thresholds, severities and escalation are actually maintained.
What you will learn
- Define model SLIs and turn them into SLOs
- Map alert severities and owners to breach types
- Balance on-call load with meaningful alerts
What is this?
Model SLOs state measurable targets: prediction latency p95, availability, drift within a band, and quality thresholds where measurable. Alerting evaluates SLIs against these targets, pages named owners on sustained breach, and reviews alerts so noise doesn't drown the signal.
Why do you need it?
- Without targets, 'monitor' produces dashboards, not decisions.
- SLOs give engineering and product a shared definition of 'good'.
- Alerting on SLO breach focuses pages on what users actually feel.
- Error budgets turn drift blips into budget accounting, not panic.
Mental model
An airline's on-time performance contract it posts publicly
Model SLOs
The airline commits to on-time within a window (SLO), measures the delay (SLI), and when the month's budget runs negative it audits its poor days. The public contract makes improvement an obligation, not a vibe.
How it works
- Define SLIs: availability, latency p50/p95, drift metric, quality metric where labels flow.
- Set SLO targets with error budgets (e.g. p95 < 120ms 99% of the time).
- Alert rules fire on sustained breaches with severity and owner.
- A review cadence rebalances thresholds as the model evolves.
- Postmortem connects SLO misses to fixes and threshold changes.
From SLI to page.
In production
- A recording SLO: p95 latency < 120ms with 99.9% availability; drift PSI < 0.2.
- A quality SLO on labeled reconciliation (AUC) with quarterly error-budget review.
- Confirmed-drift alerts page the model owner; single spikes just log a ticket.
Best practices
- Make SLOs model-specific: one size doesn't fit fraud and recommendations.
- Alert on sustained windows and confirmed trends; single spikes get tickets.
- Assign every alert an owner and a remediation runbook.
- Review the SLO set whenever the model or its data changes materially.
Common mistakes
- SLOs on paper, thresholds tuned to 'stop the noise' until nothing pages.
- Paging on every deviation — the on-call drowns and ignores real alerts.
- No budget accounting, so a model can burn a month of quality silently.
- SLOs defined once at launch and never revisited as data shifts.
Debugging tips
- Alert volume spirals — revisit thresholds relative to the current baseline; the data moved.
- An SLO breach no one cares about — confirm the SLI still reflects user impact.
- Pages at 3am for a drift blip — enforce trend-confirmation windows.
Interview checkpoint
Hands-on lab
- Define SLIs and a budgeted SLO
- Evaluate p95 over rolling windows
- Account error budget across windows
Draft and Evaluate a Model SLO
35 min · intermediate
Quick quiz
An error budget lets teams…
Model SLOs should be tuned per model and revisited when data shifts.