Monitoring
Advanced 18 min read Monitoring

SLOs, SLIs & Error Budgets

SLOs make reliability measurable and debatable. Master SLI selection, targets and windows, error budget math, burn-rate alerting, and the processes that turn budget exhaustion into engineering decisions instead of heroics.

SLISLOSLAerror budgetburn ratemulti-windowreliabilitygolden signalsSREavailability
Prerequisites: Metrics & Prometheus Alerts & Alert Fatigue
Introduction

What is this?

An SLI (service level indicator) is a quantifiable measure of a user-facing behavior, such as availability or latency. An SLO (service level objective) is a target for that SLI over a window, such as '99.9% of requests succeed and complete in under 300ms over 30 days'. The error budget is the allowed failure — everything above the target — and burning it faster than planned is the trigger for stopping risky work. Burn-rate alerting pages based on how fast the budget is consumed, not on arbitrary thresholds.

Motivation

Why do you need it?

  • SLOs force you to define what 'good' means before an incident, instead of arguing about it during one.
  • Error budgets turn reliability into a product decision: the team can spend failure budget on innovation, consciously.
  • Burn-rate alerting pages on budget consumption speed, catching both sudden outages and slow degradation with less noise.
  • Multi-window, multi-burn-rate alerting reliably separates real budget burns from short blips.
  • SLOs replace reliability debates with data conversations shared with stakeholders.
Think about it

Mental model

Imagine

A personal monthly budget

Think of

An error budget

Your SLO target is what you promise to spend, and the error budget is the money you are allowed to waste before you must stop. A burn-rate alert is the notification that you have spent 80% of the month's fun-money in the first week — the alarm is not about the amount, it is about how fast it is disappearing.

Mechanics

How it works

  • Define an SLI with an explicit good/bad definition, for example a good request is one with a status below 500 and a duration under 300ms.
  • Pick an SLO window and target, such as 99.9% availability over a rolling 30 days.
  • Compute the error budget: one minus the target, converted to real time, e.g. 0.1% of 43,200 minutes is 43.2 minutes per 30 days.
  • Track budget consumption continuously: each bad event burns budget at a rate, and the burn rate is the observed error rate divided by the allowed error rate.
  • Alert on burn rate with multi-window rules — a fast-burn window (14.4x for 5 minutes) and a slow-burn window (3x for 1 hour) — so both acute and chronic failures page.

From user behavior to an SLO-driven page.

User request
SLI
SLO
Error budget
Burn rate
Page
Reference

Key commands

curl -s 'http://localhost:9090/api/v1/query?query=error_rate_30m'

Read a recording rule that computes the SLI error rate over 30 minutes.

curl -s 'http://localhost:9090/api/v1/query?query=burn_rate'

Query the current burn rate against the SLO target.

curl -s http://localhost:9090/api/v1/rules

Inspect the recording and alerting rules backing the SLO.

promtool check rules slo.rules.yml

Validate the SLO rule file before reloading.

curl -s http://localhost:9090/api/v1/targets | grep state

Confirm the SLI source targets are being scraped.

Hands-on

Examples

SLI, error budget and burn-rate math in PromQL

SLI, error budget and burn-rate math in PromQL
# Allowed error rate implied by the SLO
1 - 0.999                      # = 0.001 → 43.2 minutes of the 43,200 in 30 days
 
# Good requests: status < 500 AND latency under 300ms
sum(rate(http_requests_total{status!~"5..", duration<=0.3}[30m]))
 
# Observed error rate over the last 30 minutes
sum(rate(http_errors_total[30m])) / sum(rate(http_requests_total[30m]))
 
# Burn rate: how fast the error budget is being consumed
sum(rate(http_errors_total[30m])) / sum(rate(http_requests_total[30m])) / (1 - 0.999)
 
# Fast-burn page: consuming at >14.4x for 5 minutes
rate(http_errors_total[5m]) / rate(http_requests_total[5m]) / (1 - 0.999) > 14.4

The budget comes from the SLO target: 0.1% of a 30-day window. Burn rate is the observed error rate divided by the allowed error rate, so 14.4x means the budget that should last 30 days would be gone in about one hour. Alerting on burn rate, rather than the raw error rate, produces the same page whether traffic is high or low.

Real world

In production

  • The checkout service carries an availability SLO of 99.9% and a latency SLO of p99 under 300ms, both measured at the API gateway.
  • When burn rate crosses the page threshold, the deployment pipeline freezes until the budget recovers — reliability gates shipping.
  • A monthly SLO review is where reliability decisions get made: what to fix, what to invest in, and what target to renegotiate.
  • Three burn-rate alerts replaced forty threshold alerts, cutting nightly noise while catching degradation earlier.
  • Per-region SLOs surface regional regressions that a global average would hide.
Guidelines

Best practices

  • Pick SLIs that reflect user-visible good and bad, measured from the front door of the service.
  • Use rolling windows — a 28 to 30 day window plus a recent short window — to catch both chronic and acute problems.
  • Set realistic targets; a 100% target guarantees constant paging and freezes all innovation.
  • Page on burn rate, not raw error rate; that is what makes the budget useful for alerting.
  • Track the error budget on a dashboard the team reads daily, not a spreadsheet updated after incidents.
Watch out

Common mistakes

  • Choosing SLIs that nobody can verify against real user impact.
  • Setting 99.999% targets when the team has no capacity or process to defend them.
  • Alerting on raw error rate so the pager fires on blips that barely touch the budget.
  • Computing SLOs only in post-hoc spreadsheets instead of continuously from metrics.
  • Treating budget exhaustion as a fire drill instead of an enforced process gate on risky changes.
Root cause analysis

Common issues

Error
A burn-rate alert fires but users seem fine
Root cause

The SLI definition counts events users do not see, or the denominator dropped, inflating the rate.

Fix

Review the good/bad definition against real user traffic and confirm request volume in the denominator.

Error
The error budget is exhausted instantly during a flappy deploy
Root cause

A fast burn on a poor rollout while no multi-window confirmation was in place.

Fix

Use canary rollouts, verify the burn with the slow-burn window, and gate deploys on budget health.

Error
The SLO dashboard shows below-target for old, irrelevant data
Root cause

The window is misaligned so old incidents still count against the rolling window.

Fix

Use aligned rolling windows evaluated from the current instant and consider a shorter recency window.

When things break

Debugging tips

  • When a burn alert fires, compute which window consumed the most budget to tell acute from chronic.
  • Verify the SLI denominator — a drop in request volume can fake a high error rate.
  • Replay SLO rules with promtool test rules against historical data to tune windows.
  • Keep separate recording rules for good and bad totals so you can inspect the inputs of the burn rate.
  • Compare the 1-hour window against the 30-day window to see whether a burn is new or long-running.
Interviews

Interview checkpoint

Practice

Hands-on lab

Compute an Error Budget and a Burn Rate
30 min · advanced
In progress
Objectives
  • Feed an SLI through node-exporter's textfile collector
  • Define recording rules for error rate and burn rate
  • Evaluate the burn rate against the 99.9% target
  • Reason about the resulting error budget in minutes

Compute an Error Budget and a Burn Rate

30 min · advanced

0/8 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/3
1

A service with a 99.9% SLO over 30 days has an error budget of approximately:

2

A burn rate of 1 means:

3

Alerting on error-budget burn rate reduces alert fatigue compared with raw error thresholds.

Quick reference

Cheat sheet

SLO math
30d window = 43,200 minBase for availability math
budget = (1 - target) × window99.9% → 43.2 min
burn rate = error rate / (1 - target)Speed of budget consumption
14.4x for 5mFast burn: 5% of budget per hour
3x for 1hSlow burn: sustained degradation
PromQL for SLOs
good_requests_totalRecording rule for SLI numerator
bad_requests_totalRecording rule for failures
error_rate_30mObserved error rate over the window
burn_rateObserved ÷ allowed error rate
promtool test rulesReplay rules against history
Go deeper

Official documentation

Google SRE Workbook — Implementing SLOs
docs
Google SRE Workbook — Alerting on SLOs
docs
Google SRE Book — Service Level Objectives
docs
Prometheus — Recording Rules
docs