Observability in Production
Know before the pager knows
Operate from telemetry, not hunches: SLIs/SLOs with error budgets, correlated traces/logs/metrics behind one query surface, and dashboards that answer 'is the service okay' before anyone has to ask.
What you will learn
- Define SLIs and SLOs per user journey
- Treat the error budget as a deployability budget
- Correlate metrics, logs, and traces in production
Observability in production means the service is understandable from telemetry: an SLO per user journey (availability, latency, correctness), an error budget that converts the tolerance into deployability currency, and traces/logs/metrics unified by request IDs/attributes so any symptom jumps to its cause in one query.
What is this?
SLIs: the measured thing per journey (e.g. 'fraction of checkouts completed <2s'), SLOs: the committed target (99.9% in the window), error budget: (1−SLO)% of time/requests you may be bad before the users feel it — spending it (or not) is a business decision that gates releases. Budget alerts trigger faster than static thresholds: 'we're 5 hours from burning the whole budget'. Correlation: trace IDs flow into logs and metrics dimensions; or even a single 'golden' query — search by trace ID, by customer, by region/lang/version — so a split is two clicks, not a scavenger hunt. Dashboards: SLO per journey, latency percentile per tier, saturation (CPU/queue), error rate; revision/version labels so 'new deploy vs old' is always a comparison axis. Standard practice: alert on symptom (SLO breach rate) not cause; alerts must be actionable (on-call runbook link). Production-level observability is a first-class deliverable with the deploy, not a 'we'll monitor tomorrow'.
Why do you need it?
- SLOs convert 'be reliable' into a measurable, budgeted commitment.
- Error budget turns reliability from a vibe into a deploy decision.
- Correlation collapses MTTR: one trace ID in a log line finds the trace and the metrics.
- Symptom-first alerts page humans for real user harm, not for a busy dashboard.
Mental model
Dashboards full of numbers, nobody knows if we're okay
A fuel gauge with a distance-to-empty readout
An error budget is distance-to-empty for your reliability.
How it works
- Instrument every user journey with a SLI (count + good + total).
- Store traces with rich attributes; add the trace ID to logs.
- Aggregate into SLO windows with budget burn alerting.
- Dashboards + one correlation query surface for triage.
From journeys to the deploy gate.
In production
- Checkout SLO 99.95% latency: 1h of budget burn per month; alert at 10% burn in 1h.
- Multi-version canary compared on the same dashboard: latency split by deploy label.
- An incident doc that starts with the trace ID, so postmortem is already half-written.
Best practices
- Alert on symptom (SLO burn) not on cause heuristics.
- Every SLI has a good/total; every alert links a runbook.
- One query surface: metrics, logs, traces correlated by trace ID.
Common mistakes
- SLOs invented for marketing, untouched for 6 months — nobody trusts them.
- Dashboards that are monitors, not journeys: hundreds of panels, no verdict.
- Alerts for cause-flavored things (queue length > X) that fire when the users are fine.
Debugging tips
- Slow checkout for a customer → open the trace, read the span list; the expensive span is the bug.
- SLO burning but dashboards green → your SLI is missing the 'good' count (e.g. not counting latency>X as bad).
Interview checkpoint
Quick quiz
Alert on symptom or cause?