Observability
Intermediate 11 min read Observability

Alerting and SLOs for AI

Objectives you can wake someone for

SLOs for AI bundle latency, error rate, cost and quality into service-level objectives, with burn-rate alerting tuned so minor blips don't page anyone but real degradations do. Budget alerts with error budgets, not just thresholds.

SLOSLIalertingburn rateerror budget
Prerequisites: Evaluation in Production
Before we start

What you will learn

  • Define SLIs for AI (latency, errors, cost, quality)
  • Set SLOs with achievable targets
  • Tune burn-rate alerts on error budgets
Introduction

What is this?

An SLO is a target over time (e.g. 99.5% of requests with a judged-usable answer in under 8s). SLIs are the measured indicators each objective wraps. Alerting fires on error-budget burn — how fast you're spending this month's allowed badness — rather than on every threshold wiggle.

Motivation

Why do you need it?

  • Latency flapping is noise; error-budget burn is signal.
  • AI errors are fuzzy — a wrong answer isn't an HTTP 500, so quality needs its own SLI.
  • Cost per request deserves an SLO too (silent overspend is a support case pattern).
Think about it

Mental model

Imagine

SLOs

Think of

A household's electricity budget

You don't alert on one dim second (single blip); you alert when the month's consumption outpaces the plan — burn rate.

Mechanics

How it works

  • Define SLIs: p95 TTFT, error rate, judge-quality pass rate, cost per 1K tokens.
  • Set SLO targets with a realistic error budget (don't promise 99.99% when prompts vary).
  • Measure burn rate: how fast the current month's budget is consumed.
  • Alert on burn windows (e.g. 14.4% in 1h) instead of static thresholds.

Burn-rate alert anatomy.

SLI
Budget
Burn
Alert
Real world

In production

  • Chat SLO: p95 answer < 6s, quality pass ≥ 97%; burn alerts page on either.
  • Batch SLO: cost per 1K tokens within 10% of model forecast.
  • Agent SLO: step success ≥ 95% with burn-rate alerting on tool failures.
Guidelines

Best practices

  • Three SLIs is plenty; pick latency, quality, cost.
  • Set targets from measured baselines, then tighten.
  • Alert on burn windows and let dashboards show the trend.
  • Document what each alert means and the runbook action.
Watch out

Common mistakes

  • Paging on any p95 blip (static threshold alerts).
  • SLO targets that are impossible or free (99.99% on stochastic prompts).
  • No quality SLI — a bug-silent product can still hit its latency SLO.
When things break

Debugging tips

  • Alert overload → convert static rules to burn-rate; fewer, meaningful pages.
  • Quality SLO failing but latency fine → prompt/model issue; check version deltas.
  • Cost SLI exceeded → tokens/min up or price model mix; split by model dimension.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Burn-rate alerting pages you when?

2

A quality SLI is optional if your latency is great.

Go deeper

Official documentation

Google SRE workbook
docs
Burn rate alerting
docs