Topic 11 of 12

Observability and incident response

Learn to see inside a running system with metrics, logs and traces, set alerts and SLOs, and respond to incidents with a local Prometheus and Grafana lab.

After this topic you can: You can instrument a small service, query and chart its metrics, alert on user-facing symptoms, and run a calm incident from detection to a written postmortem.

8 lessons · 92 min · 0 of 8 done

Best read after: Containers with Docker, Kubernetes

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Lessons, in order

  1. 1Monitoring vs observability: metrics, logs, tracesCoreLearn what monitoring and observability mean, what metrics, logs and traces each tell you, and see all three from a tiny local app.10 min
  2. 2Metrics with Prometheus: scraping and PromQLCoreUnderstand how Prometheus pulls metrics from a /metrics page, how counters differ from gauges, and run your first PromQL queries.14 min
  3. 3Logs that help: structure, levels, correlation IDsCoreLearn how structured logs, levels and a request ID turn a pile of text into something you can search, and where Loki fits in.9 min
  4. 4Traces and OpenTelemetry: following one requestAdvancedUnderstand spans, traces and context propagation, what OpenTelemetry is, and generate your own spans locally with no backend.12 min
  5. 5Dashboards with Grafana: charts that answer questionsCoreConnect Grafana to Prometheus and build a small dashboard around traffic, errors, latency and saturation that you can read in seconds.10 min
  6. 6Alerting and SLOs: paging on what users feelCoreLearn SLIs, SLOs and error budgets as Google's SRE material defines them, then write a Prometheus alert on a symptom and watch it fire.15 min
  7. 7Incident response: triage, communicate, mitigateCoreLearn the roles and order of work in an incident, then practise detect, triage, mitigate and log a timeline on a deliberately broken lab app.13 min
  8. 8Blameless postmortems: learning after the incidentCoreLearn what a good postmortem contains, why blameless wording works, and write one from your lab incident using a simple template.9 min

Keep these open while you learn

Next topic

Security basics for DevOpsLearn the security habits a DevOps engineer applies daily: threat thinking, secrets, least privilege, scanning, supply chain, pipelines and TLS.

Search

Search cheat sheets and projects