Observability
Intermediate 14 min read Observability

Alerting, On-Call & Runbooks

When the page fires, someone has an answer

The last mile: alert delivery (escalation, dedupe, routing), on-call rotations (the human side: coverage, handoffs, availability), and runbooks (the step-by-step that turns a page into a fix). Blameless postmortems close the loop.

oncallescalationrunbookPagerDutyOpsgenierotationpostmortem
Prerequisites: Metrics & Alerting Dashboards & Visualization
Before we start

What you will learn

  • Design alert routing, dedupe, escalation, and severities
  • Run sustainable on-call rotations
  • Write runbooks and blameless retrospectives
In 60 seconds

Alerts are the trigger; the machine (page service: pager-duty/opsgenie/cloudwatch-notifications) routes + escalates; humans on rotation receive. Runbooks hold the before-the-brain answer; postmortems convert each incident into guardrails.

Introduction

What is this?

Delivery: PagerDuty/Opsgenie/Slack/Teams + provider's Amazon Simple Notification + webhooks; dedupe keys (same alert grouped); escalation chains (primary → secondary → on-duty management); rotations (weekly, RFC 2350-ish handoffs), on-call handoff briefings. Severities: P1 (critical/user-impacting fire), P2 (degraded), P3 (notice). Silencing/maintenance + overrides. Runbooks: per alert/severity: prerequisites, first commands, restore path, contacts; git-versioned (runbooks-as-code). Incidents: war room, timeline, impact statement, resolution, blameless postmortem + action item tracker (trackers, automation from ticket to guardrails). Fresh-eyes review quarterly + audit.

Motivation

Why do you need it?

  • A page without a runbook is a pager and a hope; a page with one is a checklist.
  • Escalation + dedupe prevent the 2am 'page storm into silence' classic.
  • Sustainable rotations (handoff, time-off, blameless culture) avoid burnout → opseople quit.
  • Postmortems convert one costly incident into a config/policy change → the whole system improves.
Think about it

Mental model

Imagine

A fire station: alarm (alert), crew line-up (rotation), engine checklist by model (runbook), and an after-action drill review (postmortem)

Think of

Alerting + on-call system

The alarm alone saves nothing; the lineup (who's on), the checklist (what to do), and the drill-after (improve) make the team effective. Only installing an alarm is the classic failure.

Mechanics

How it works

  • Alert fires → page service dedupes → routes by severity/route (service/env).
  • Primary on-call is paged; escalation + everyone-sleep rule if ack'd not in N min.
  • On-call follows the runbook (commands + restore path) with a live war room.
  • Incident closed → timeline + impact captured → blameless postmortem within 48h.
  • Postmortem actions (guardrail, alert fix, runbook update) tracked to closure.

Alert → fix → improve.

Alert fires
Page service
On-call
Restore
Postmortem
Hands-on

Examples

Runbook skeleton (git-versioned)

Runbook skeleton (git-versioned)
---
name: "High load P1"
severity: p1
service: api
steps:
  - name: Confirm scope
    command: "grep -c '5..' trace; kubectl get po -l app=api"
  - name: Triage
    command: "curl /metrics | grep http_rest"
  - name: Restore path
    command: "scale: kubectl scale deploy/api --replicas=6 | rollback image: kubectl rollout undo deploy/api"
escalation: pay-api-oncall → platform-lead
post_action: update trailing metrics in the SLO dashboard

A runbook is reviewable like code — every alert's page carries the URL, so the on-call never improvises.

Real world

In production

  • Alert routing by team/severity into one page service (dedupe key + escalation).
  • Weekly handoff: 30-min briefing of changes, pending runbooks, active experiments.
  • Postmortems auto-created from the repo template with action tracking.
Guidelines

Best practices

  • Every alert must carry a runbook URL — no bare pages.
  • Dedupe + escalation tuned so 2am page storms collapse into one actionable call.
  • On-call is a job, not a martyrdom: handoffs, time-off, and rotation reviews matter.
  • Keep runbooks current (they go stale): runbook hygiene = incident-readiness.
  • Blameless retro: timeline + impact + actions; the 'who' never matters.
Watch out

Common mistakes

  • Paging on-cause-alerts (a pod restart) instead of symptoms (user-visible).
  • Dedupe missing → '24 notifications for one root cause' noise training.
  • Runbooks written once, never updated — the fix described no longer exists.
  • Postmortem blaming people → next incident gets hidden instead of shared.
When things break

Debugging tips

  • Pages that never ack → verify rotation metadata + phone/app config; test a drill page.
  • Duplicate storm → fix the alert's grouping/dedupe key, not the code.
  • Runbook missing in page → enforce the link at rule creation (CI lint your alert YAML).
  • Action items lingering → quarterly incident review reconciles open trackers.
Interviews

Interview checkpoint

Practice

Hands-on lab

Run a Synthetic Incident with a Runbook
30 min · intermediate
In progress
Objectives
  • Trigger a synthetic degrade and confirm the page fires
  • Walk the runbook to the restore step
  • File the blameless postmortem + action

Run a Synthetic Incident with a Runbook

30 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The single best sanity rule for pages is:

2

A postmortem's deliverable is:

Go deeper

Official documentation

PagerDuty reps
docs
Google SRE: on-call
docs
Amazon CloudWatch notifications
docs