Lesson 6 of 8 in Observability and incident response
Alerting and SLOs: paging on what users feel
A service level indicator (SLI) is a carefully defined measure of one aspect of service quality, such as the share of requests that succeed. A service level objective (SLO) is a target for an SLI, such as 95 percent over a period. The error budget is the amount of failure the SLO still allows.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Metrics with Prometheus: scraping and PromQL, Dashboards with Grafana: charts that answer questions
The problem this solves
Without a target, every blip looks equally serious. A team either ignores alerts because most are noise, or chases every small wobble and burns out. Neither protects users.
An SLO fixes a shared, written line for how good is good enough, measured from the user's side. Perfect reliability is not a sensible target, since it is extremely costly and users cannot tell the difference at some point. The gap between the SLO and 100 percent is the error budget, which can be spent on releases and experiments.
Alerts follow from the SLO. Google's SRE book advises paging on symptoms, which are things users feel such as failing requests, and using causes, such as a busy CPU, mainly as debugging aids. A good page is urgent, actionable and needs human judgment.
How it works
- Other
- Control
- Data
- Security
An SLI is measured, an SLO sets the target for it, the error budget is what remains, and an alert fires when the budget is being used up too fast.
- SLI to SLO: 1 compare to target
- SLO to Error budget: 2 100% minus target
- Error budget to Alert: 3 burn rate too high
- Other
- Control
- Security
- Network
A rule starts inactive, becomes pending when its expression is true, becomes firing after staying true for the configured time, and then goes to a notifier.
- Inactive to Pending: 1 expression true
- Pending to Firing: 2 true for `for` time
- Firing to Alertmanager: 3 send alert
Choose an SLI from what users care about, for example the share of requests that succeed. The SLO is a target for it, over a stated window. If the SLO is 95 percent, the error budget is the remaining 5 percent of requests that are allowed to fail in that window.
Burn rate is how fast you use the budget compared with the pace that would use it up exactly at the end of the window. A burn rate of 1 uses the whole budget by the end of the window, and 2 uses it twice as fast. In the lab the error ratio divided by 0.05 is the burn rate for a 95 percent SLO.
An alert rule in Prometheus is a PromQL expression plus a for duration. While the expression is true the alert is pending, and after it has stayed true for the for time it is firing. The for time filters out brief blips. Prometheus only evaluates rules and shows their state. Sending an email or chat message needs Alertmanager, which this lab leaves out.
The lab uses a one minute window so you can see results quickly. Production alerts usually combine longer and shorter windows, so a slow steady burn is caught as well as a sudden one.
Try it yourself
Continue in ~/obs-lab. You will define a 95 percent success SLO on paper, load an alert that fires when the budget burns at more than twice the normal pace, then break the app on purpose to watch it fire.
- 1Look at the SLI in Prometheus
# paste into http://localhost:9090/graph sum(rate(app_requests_total{status="ok"}[5m])) / sum(rate(app_requests_total[5m]))This is the SLI, the share of successful requests. With the default 2 percent failure it should hover around 0.98, above the 0.95 SLO.
- 2Add an alert rule
cd ~/obs-lab cat > prom/alerts.yml <<'EOF' groups: - name: demo rules: - alert: ErrorBudgetBurningFast expr: (sum(rate(app_requests_total{status="error"}[1m])) / sum(rate(app_requests_total[1m]))) / 0.05 > 2 for: 1m labels: severity: page annotations: summary: "Failing requests are using the error budget more than twice as fast as allowed" EOF cat > prom/prometheus.yml <<'EOF' global: scrape_interval: 5s rule_files: - /etc/prometheus/alerts.yml scrape_configs: - job_name: demo-app static_configs: - targets: ["app:8000"] EOFThe expression divides the error ratio by 0.05, the budget for a 95 percent SLO, so a value above 2 means the budget is burning at more than twice the allowed pace.
- 3Check the rule file and reload
docker run --rm -v "$PWD/prom:/p:ro" --entrypoint promtool prom/prometheus check rules /p/alerts.yml docker compose restart prometheus
promtoolships in the Prometheus image and reports syntax errors before you load the file. Then openhttp://localhost:9090/alerts. The rule should be listed as inactive. - 4Break the app on purpose
ERROR_RATE=0.2 docker compose up -d app
Compose recreates the app with a 20 percent failure rate. Refresh the alerts page every half minute. The state goes from inactive to pending, then to firing after the one minute
fortime. - 5Restore the normal rate
ERROR_RATE=0.02 docker compose up -d app
After the one minute window fills with healthy data, the alert returns to inactive.
What breaks, and where to look
The alert does not appear on
/alerts, or the Rules page is empty.- WHY IT HAPPENS
- The rules file was not loaded. The
rule_filespath may be wrong, the YAML may be invalid, or Prometheus was not restarted. - CHECK FIRST
- Open
http://localhost:9090/rules, then rundocker compose logs prometheusand read the last lines for a rule or config error.
The alert flaps between firing and inactive, or pages you for a few seconds of errors.
- WHY IT HAPPENS
- There is no
forduration, or therate()window is very short, so ordinary noise crosses the threshold. - CHECK FIRST
- Graph the alert's expression in
http://localhost:9090/graphover the last hour and add afortime or a longer window so ordinary noise stays below the line.
The app is completely down, yet no alert fires and the error ratio panel says
No data.- WHY IT HAPPENS
- With no requests at all, the ratio has no value. Dividing by nothing produces an empty result, and an empty result is not greater than 2.
- CHECK FIRST
- Add a second rule on
up == 0or onabsent(app_requests_total)so a service that stops reporting still raises an alert.
When not to use this
- A tiny internal tool with no real users may not need a formal SLO. A simple uptime check and a note on who to ask is enough.
- Do not page on every cause, such as CPU above 80 percent. If users are unaffected, send it to a dashboard or ticket instead, because pages that need no action train people to ignore pages.
- An SLA is a contract with consequences if missed, while an SLO is an internal target. Do not promise an SLA until you have measured an SLO for a while.
Check yourself
If the SLO is 99 percent successful requests, what is the error budget?
1 percent of requests may fail in the measurement window.
Why page on a symptom such as failing requests rather than on high CPU?
Symptoms are what users feel and are urgent by nature. High CPU may be harmless, and is better used as a hint when debugging.
What does the for field in a Prometheus alert rule do?
It requires the expression to stay true for that long before the alert moves from pending to firing, which filters short blips.
Keep going
Lesson 83 of 92 in the DevOps path