Dashboards & Visualization
The 3am first-glance cockpit
Dashboards are the team's shared nervous system: RED/USE panels per service, SLO burn panels, dependency/FLAG views. Design for the person on call at 3am — answer 'is it broken, how bad, where' inside 5 seconds.
What you will learn
- Design a single-service RED dashboard
- Layer SLO burn + dependency tiles
- Use templating + annotations for useful views
A dashboard is a set of linked panels answering: rate, errors, duration (RED), resources (USE). Good dashboards start from an incident question, get templated per service/env, and include annotations (deploys, incidents) so the 3am brain can correlate.
What is this?
Tools: Grafana (Prometheus/Loki + cloud sources), CloudWatch dashboards + widgets, Azure workbooks/Dashboards, GCP Monitoring dashboards. Building blocks: time series panels (metric-typed), stat tiles (top K), tables, logs panels, geo/heatmaps, histograms. Layout: header SLO/burn tiles → RED rows → USE resources → DAG/oplog. Templating + variables (env/service/pod) keep one board for N services. Annotations for deploys/alerts/on-calls give the 'was that deploy the problem?' answer. Alert links: each panel should deep-link to its underlying query. Culture: dashboard reviews + cleanup; orphaned boards kill signal.
Why do you need it?
- Dashboards compress a 4-hour investigation into a first-glance 'which panel went red'.
- Shared cockpit aligns the team on what healthy looks like.
- Annotations tie incident timeline to deploys/metric dips automatically.
- Templating means you maintain one board, not one per service.
Mental model
An aircraft cockpit's top row: alt/airspeed/engine gauges framed for glanceability
Red dashboard
The pilot doesn't read a 200-widget wall; the critical five gauges sit for a 1-second scan. Fixing the coffee-dash of hundreds of tiles is the whole art.
How it works
- Instrument FIRST; then define SLIs; then arrange panels that answer the SLOs.
- Each panel queries a metric/log with templating vars; annotations overlay events.
- Click-through per panel → underlying query/alert for drill-down.
- Refresh cadence sized to the data (1m for hot, 15m for SLO-burn).
- Boards shared/versioned; alert links keep the story coherent.
Dashboard anatomy.
Examples
SLO burn panel (Grafana-style)
- targets: [{ expr: '1 - sum(rate(http_requests_total{job="api",status=~"5.."}[1h])) / sum(rate(http_requests_total{job="api"}[1h]))' }] title: 'SLO availability (99.9%)' thresholds: { steps: [ { color: 'green', value: 99.9 }, { color: 'orange', value: 99.0 }, { color: 'red', value: 90 } ] }A stat tile colored by SLO threshold — the 3am answer to 'are we bleeding budget' in one glance.
In production
- Per-service RED board (templated by $service), link from each pager alert.
- Used by-burn panel pinned in the ops war-room wall.
- Dependency board (traffic/dB/queue depths) next to the service dashboards.
Best practices
- Start each dashboard from a real incident question; nothing abstract.
- Keep dashboards ≤ ~15 panels; add drill-down links not more panels.
- Template by env/service to avoid copy-paste boards.
- Annotate deploys + incidents automatically (CI hook).
- Retire boards quarterly; dead signal is worse than no board.
Common mistakes
- Dashboard full of panels nobody reads (orphan clickbait tiles).
- Panels with different time windows/mistune units that mislead at-a-glance.
- No annotations: 'was the deploy the problem' stays a debate.
- Grafana/graph per-engineer → fragmented truth + alert-to-Dash mismatches.
Debugging tips
- A panel looks empty → check the metric name + label selectors for the chosen $env.
- Two dashboards disagree → both point at different queries/aggregations; unify via shared dashboard lib.
- Delay between action and panel movement → scrape interval/refresh cadence mismatch.
- Annotation missing a deploy → CI hook env vars (tag/branch) not passed to the API call.
Interview checkpoint
Quick quiz
RED stands for:
The right first move when a board mis-leads: