Lesson 6 of 8 in Observability and incident response

Alerting and SLOs: paging on what users feel

Core15 min

A service level indicator (SLI) is a carefully defined measure of one aspect of service quality, such as the share of requests that succeed. A service level objective (SLO) is a target for an SLI, such as 95 percent over a period. The error budget is the amount of failure the SLO still allows.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Metrics with Prometheus: scraping and PromQL, Dashboards with Grafana: charts that answer questions

The problem this solves

Without a target, every blip looks equally serious. A team either ignores alerts because most are noise, or chases every small wobble and burns out. Neither protects users.

An SLO fixes a shared, written line for how good is good enough, measured from the user's side. Perfect reliability is not a sensible target, since it is extremely costly and users cannot tell the difference at some point. The gap between the SLO and 100 percent is the error budget, which can be spent on releases and experiments.

Alerts follow from the SLO. Google's SRE book advises paging on symptoms, which are things users feel such as failing requests, and using causes, such as a busy CPU, mainly as debugging aids. A good page is urgent, actionable and needs human judgment.

How it works

From a measurement to a page
From a measurement to a pageAn SLI is measured, an SLO sets the target for it, the error budget is what remains, and an alert fires when the budget is being used up too fast.1 compare totarget2 100% minustarget3 burn ratetoo highSLIgood requests / allSLOtarget, e.g. 95%Error budget5% may failAlertbudget burning fast
  • Other
  • Control
  • Data
  • Security

An SLI is measured, an SLO sets the target for it, the error budget is what remains, and an alert fires when the budget is being used up too fast.

  1. SLI to SLO: 1 compare to target
  2. SLO to Error budget: 2 100% minus target
  3. Error budget to Alert: 3 burn rate too high
How a Prometheus alert changes state
How a Prometheus alert changes stateA rule starts inactive, becomes pending when its expression is true, becomes firing after staying true for the configured time, and then goes to a notifier.1 expressiontrue2 true for`for` time3 send alertInactiveexpression is falsePendingtrue, waiting for:Firingtrue long enoughAlertmanagerroutes notifications
  • Other
  • Control
  • Security
  • Network

A rule starts inactive, becomes pending when its expression is true, becomes firing after staying true for the configured time, and then goes to a notifier.

  1. Inactive to Pending: 1 expression true
  2. Pending to Firing: 2 true for `for` time
  3. Firing to Alertmanager: 3 send alert

Choose an SLI from what users care about, for example the share of requests that succeed. The SLO is a target for it, over a stated window. If the SLO is 95 percent, the error budget is the remaining 5 percent of requests that are allowed to fail in that window.

Burn rate is how fast you use the budget compared with the pace that would use it up exactly at the end of the window. A burn rate of 1 uses the whole budget by the end of the window, and 2 uses it twice as fast. In the lab the error ratio divided by 0.05 is the burn rate for a 95 percent SLO.

An alert rule in Prometheus is a PromQL expression plus a for duration. While the expression is true the alert is pending, and after it has stayed true for the for time it is firing. The for time filters out brief blips. Prometheus only evaluates rules and shows their state. Sending an email or chat message needs Alertmanager, which this lab leaves out.

The lab uses a one minute window so you can see results quickly. Production alerts usually combine longer and shorter windows, so a slow steady burn is caught as well as a sudden one.

Try it yourself

Continue in ~/obs-lab. You will define a 95 percent success SLO on paper, load an alert that fires when the budget burns at more than twice the normal pace, then break the app on purpose to watch it fire.

  1. 1
    Look at the SLI in Prometheus
    # paste into http://localhost:9090/graph
    sum(rate(app_requests_total{status="ok"}[5m])) / sum(rate(app_requests_total[5m]))

    This is the SLI, the share of successful requests. With the default 2 percent failure it should hover around 0.98, above the 0.95 SLO.

  2. 2
    Add an alert rule
    cd ~/obs-lab
    cat > prom/alerts.yml <<'EOF'
    groups:
      - name: demo
        rules:
          - alert: ErrorBudgetBurningFast
            expr: (sum(rate(app_requests_total{status="error"}[1m])) / sum(rate(app_requests_total[1m]))) / 0.05 > 2
            for: 1m
            labels:
              severity: page
            annotations:
              summary: "Failing requests are using the error budget more than twice as fast as allowed"
    EOF
    cat > prom/prometheus.yml <<'EOF'
    global:
      scrape_interval: 5s
    rule_files:
      - /etc/prometheus/alerts.yml
    scrape_configs:
      - job_name: demo-app
        static_configs:
          - targets: ["app:8000"]
    EOF

    The expression divides the error ratio by 0.05, the budget for a 95 percent SLO, so a value above 2 means the budget is burning at more than twice the allowed pace.

  3. 3
    Check the rule file and reload
    docker run --rm -v "$PWD/prom:/p:ro" --entrypoint promtool prom/prometheus check rules /p/alerts.yml
    docker compose restart prometheus

    promtool ships in the Prometheus image and reports syntax errors before you load the file. Then open http://localhost:9090/alerts. The rule should be listed as inactive.

  4. 4
    Break the app on purpose
    ERROR_RATE=0.2 docker compose up -d app

    Compose recreates the app with a 20 percent failure rate. Refresh the alerts page every half minute. The state goes from inactive to pending, then to firing after the one minute for time.

  5. 5
    Restore the normal rate
    ERROR_RATE=0.02 docker compose up -d app

    After the one minute window fills with healthy data, the alert returns to inactive.

What breaks, and where to look

  • The alert does not appear on /alerts, or the Rules page is empty.

    WHY IT HAPPENS
    The rules file was not loaded. The rule_files path may be wrong, the YAML may be invalid, or Prometheus was not restarted.
    CHECK FIRST
    Open http://localhost:9090/rules, then run docker compose logs prometheus and read the last lines for a rule or config error.
  • The alert flaps between firing and inactive, or pages you for a few seconds of errors.

    WHY IT HAPPENS
    There is no for duration, or the rate() window is very short, so ordinary noise crosses the threshold.
    CHECK FIRST
    Graph the alert's expression in http://localhost:9090/graph over the last hour and add a for time or a longer window so ordinary noise stays below the line.
  • The app is completely down, yet no alert fires and the error ratio panel says No data.

    WHY IT HAPPENS
    With no requests at all, the ratio has no value. Dividing by nothing produces an empty result, and an empty result is not greater than 2.
    CHECK FIRST
    Add a second rule on up == 0 or on absent(app_requests_total) so a service that stops reporting still raises an alert.

When not to use this

  • A tiny internal tool with no real users may not need a formal SLO. A simple uptime check and a note on who to ask is enough.
  • Do not page on every cause, such as CPU above 80 percent. If users are unaffected, send it to a dashboard or ticket instead, because pages that need no action train people to ignore pages.
  • An SLA is a contract with consequences if missed, while an SLO is an internal target. Do not promise an SLA until you have measured an SLO for a while.

Check yourself

If the SLO is 99 percent successful requests, what is the error budget?

1 percent of requests may fail in the measurement window.

Why page on a symptom such as failing requests rather than on high CPU?

Symptoms are what users feel and are urgent by nature. High CPU may be harmless, and is better used as a hint when debugging.

What does the for field in a Prometheus alert rule do?

It requires the expression to stay true for that long before the alert moves from pending to firing, which filters short blips.

Lesson 83 of 92 in the DevOps path

Search

Search cheat sheets and projects