Observability
Intermediate 14 min read Observability

Metrics & Alerting

Numbers that page you when they lie sideways

Metrics are the fastest path to 'it's broken' — counters, gauges, histograms plus rulers that turn thresholds into pages. Alert design is the craft: fire on symptom burn-rates, not on might-y-have-issues noise.

metriccounterhistogramalertthresholdburn ratePrometheusCloudWatch
Prerequisites: Observability Foundations
Before we start

What you will learn

  • Choose counter/gauge/histogram per measurement
  • Write threshold + burn-rate alerts
  • Calibrate alert severity and routing to avoid noise fatigue
In 60 seconds

Counters (monotonic, compute rates), gauges (current value: CPU, pool), histograms (latency percentiles). Alert rules evaluate a PromQL/CloudWatch/Log query and fire when the result stays bad long enough — sleep by burn-rate windows beats a hard threshold hit.

Introduction

What is this?

Metric types: counter (inc), gauge (set), histogram (bucket → quantiles), summary. Sources: node_exporter/app metrics/CloudWatch/Provider metrics (CPU, I/O). Ruler taxonomy: prometheus rules, CloudWatch alarms, Azure alert rules, GCP alerting policies. Alert states: OK / pending / firing / resolved. Thresholds + windows: for N duration. Burn-rate SLO alerting: page when error budget burns faster than X% over a window (both multi- and single-window). Silencing/muting, severity (P1/P2/P3), action routing (Opsgenie/PagerDuty/Slack), dedupe keys. Dashboards ↔ alerts should tell the same RED/USE story.

Motivation

Why do you need it?

  • Faster than humans: an alarm fires in seconds, humans argue for minutes.
  • Correct alert design beats more alerts — noise trains brains to ignore.
  • Burn-rate-based alerts proactively catch SLO de-routes before they're 42 minutes late.
  • Counter/histogram misuse makes every graph a lie (the #1 rookie metric bug).
Think about it

Mental model

Imagine

A house with smoke detectors only in the kitchen

Think of

Bad alert coverage

You'd wire every room (every SLI) with calibrated detectors — loud for vital, silent for trivial — not just the one room you remember. Detectors all in one place = rest of the house burns silently.

Mechanics

How it works

  • App increments counters/set gauges + records histograms per request path.
  • Collector scrapes each interval; rule engine evaluates queries.
  • A rule stays in firing state only while the condition holds (Windows).
  • Burn-rate rules map error budget consumption to page urgency.
  • Alerts dedupe + route → on-call with a runbook link.

Metric → alert journey.

Instrument
Collector
Rule
States
Route
Hands-on

Examples

Burn-rate alert (10% budget/hr)

Burn-rate alert (10% budget/hr)
# 99.9% SLO, page if we'd exhaust 10% in 1h => 99% over the hour
rule:
  expr: |
    (1 - sum(rate(http_requests_total{status=~"5.."}[1h])) /
          sum(rate(http_requests_total[1h]))) < 0.99
  for: 15m
  labels: { severity: "p1", slo: "http-availability" }

A sustained burn-rate alert: the 15m 'for' removes flappy noise; the burn math makes the page correspond to SLO risk, not a whim.

Real world

In production

  • RED dashboards per service + P1 burn-rate on the central SLO.
  • Gauge alert on node memory saturation (swap imminent) with a runbook.
  • Histogram latency alerts per endpoint percentile with window smoothing.
Guidelines

Best practices

  • Alert on symptoms (latency/error/burn) not causes (a single pod OOM).
  • Page on burn-rate; notify loudly only those with authority to act.
  • Every alert has: severity, owner, runbook, mute path, and an end-to-end test.
  • Keep alert count per service small (5–15 real ones).
  • Test the alert pipeline: fire a synthetic incident monthly.
Watch out

Common mistakes

  • alerting on gauge CPU of a single pod (thrash city).
  • Thresholds far from actual capacity → 'page storms' at 2x design.
  • Histogram sampled wildly → percentiles mean nothing.
  • No mute/README → on-call inherits an alarm and guesses.
When things break

Debugging tips

  • Alert fires for 1 second then OK → the 'for' window is too short or reset logic.
  • Burn-rate alert always on → your SLO target and burn math disagree; re-evaluate windows.
  • Outlier noisy series → break by a larger label (job+route), not a global sum.
  • Silence blind spot → check the mute list/README before blaming the rule.
Interviews

Interview checkpoint

Practice

Hands-on lab

Write, Fire, and Mute an Alert
30 min · intermediate
In progress
Objectives
  • Stand a local Prometheus + a tiny exporter
  • Create a threshold alert + watch states
  • Mute it intentionally and verify silence

Write, Fire, and Mute an Alert

30 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

To measure p99 latency you use:

2

The purpose of the 'for' duration is:

Go deeper

Official documentation

Prometheus alerting
docs
CloudWatch alarms
docs
Azure Monitor alerts
docs