Monitoring
Advanced 17 min read Monitoring

Dashboards & Grafana

A dashboard is only useful if it drives action. Learn panel types, template variables, data sources, annotations and provisioning, and how to design dashboards that an on-call engineer can read in one glance.

Grafanadashboardpanelstemplate variablesdata sourceannotationsprovisioningPromQLactionablegolden signals
Prerequisites: Metrics & Prometheus SLOs, SLIs & Error Budgets
Introduction

What is this?

Grafana is a visualization platform that connects to data sources — Prometheus, Loki, Elasticsearch, Tempo — and renders queries into panels. A dashboard is a layout of panels that answers a question. Template variables let one dashboard serve many services, annotations overlay events like deploys on time series, and provisioning keeps dashboards in git as code. Good dashboard design starts from the question an engineer must answer, not from available data.

Motivation

Why do you need it?

  • A well-designed dashboard cuts mean-time-to-respond by showing the right signal at the top of the screen.
  • Template variables turn one dashboard into a hundred views, one per service or region.
  • Annotations correlate deploys and incidents to metric changes in the same chart.
  • Grafana unifies metrics, logs and traces behind one pane of glass instead of five tools.
  • Actionable dashboards drive decisions; decorative ones waste the attention of everyone who opens them.
Think about it

Mental model

Imagine

An airplane cockpit

Think of

A good dashboard

An airplane cockpit clusters instruments by the task at hand: one cluster for engine state, one for navigation, one for warnings. It has no Christmas tree of blinking lights. Every gauge exists because a pilot must decide something from it. A dashboard full of unrelated pretty charts is the Christmas tree — it glows, and it teaches everyone to ignore it.

Mechanics

How it works

  • A data source defines how Grafana talks to a backend, for example a Prometheus instance using its HTTP query API.
  • Each panel runs one or more queries and renders the result as a time series, stat, table, gauge or log view.
  • Template variables like $service are substituted into panel queries at render time, so one panel serves every value.
  • Annotations read events from a data source or the API and draw vertical markers on time-series panels.
  • Dashboards are versioned JSON; provisioning folders let teams ship them from git with datasources and alerting config.

How a Grafana panel turns a query into an answer.

Data source
Panel query
Render
Template variable
Annotation
Engineer
Reference

Key commands

curl -s -u admin:admin 'http://localhost:3000/api/search?type=dash-db'

List dashboards through the Grafana HTTP API.

curl -s 'http://localhost:9090/api/v1/query?query=up'

Verify the Prometheus data source backend is reachable.

kubectl get pods -l app=grafana

Locate the Grafana deployment in a cluster.

docker ps --filter name=grafana

Find the Grafana container and its port mapping.

grafana-cli plugins install grafana-piechart-panel

Install an optional visualization plugin.

Hands-on

Examples

A panel query and the provisioning file behind it

A panel query and the provisioning file behind it
// Stat panel: p99 latency for the selected service
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{service="$service"}[5m]))
)
 
// Time series panel: 5xx error rate per instance
sum by (instance) (
  rate(http_requests_total{service="$service", status=~"5.."}[5m])
)

The same dashboards JSON, provisioned from git, works for every service because $service is a template variable: the dropdown feeds a value into the query at render time. The first query computes a latency percentile; the second breaks error rate down by instance. Both stay cheap because they aggregate with sum.

Real world

In production

  • A golden-signals dashboard is the default landing page for every service: latency, traffic, errors and saturation at the top.
  • Per-service dashboards use template variables, so one JSON drives all services and new services get dashboards for free.
  • Deploy annotations on the latency panel explain a latency drop immediately after a release during incident review.
  • Grafana alerting evaluates latency-based SLO burn rules that would otherwise need a second tool.
  • Provisioning keeps dashboards in git, reviewed like code, with datasources configured per environment.
Guidelines

Best practices

  • Put one primary metric per panel; cluster related series with intent, not everything into one chart.
  • Use template variables for service, instance and region instead of hardcoding values in queries.
  • Keep a dashboard to a single screen height so it can be scanned in one glance.
  • Annotate deploys, releases and config changes on the time-series panels that matter.
  • Design every dashboard to answer a question or drive a decision; delete the ones that answer nothing.
Watch out

Common mistakes

  • Building a forty-panel 'everything' dashboard that takes ten minutes to parse during an incident.
  • Hardcoding service names in queries, forcing a duplicate dashboard per service.
  • Omitting units and thresholds so the numbers are meaningless at a glance.
  • Overlapping series with no legend, thresholds or clear naming.
  • Making dashboards that look nice but nobody is empowered to act on.
Root cause analysis

Common issues

Error
A panel shows No data
Root cause

The query selector matches nothing, or the panel uses the wrong data source.

Fix

Run the query in Explore, inspect the real label values, and confirm the panel's data source is the one with the data.

Error
The template variable dropdown is empty
Root cause

The variable's query returns no values, often because it queries the wrong data source or format.

Fix

Edit the variable, check its query and data source, and test the same query in Explore.

Error
Deploy annotations do not appear on the chart
Root cause

The annotation query filters on tags that the deploy events do not carry.

Fix

Verify the annotation data source and tags match the events, or query annotations without filters to confirm data exists.

When things break

Debugging tips

  • Iterate on a query in Grafana Explore before committing it to a panel.
  • When a panel shows No data, check its data source and query selector — the data may exist under different labels.
  • Verify template variables have defaults; an empty dropdown usually comes from a variable query with the wrong data source.
  • Check the dashboard's time range and refresh rate for stat panels; a stale 'no change' can be a refresh problem.
  • Inspect the Panel JSON when a provisioned dashboard behaves differently from one built in the UI.
Interviews

Interview checkpoint

Practice

Hands-on lab

Provision a Grafana Dashboard with a Prometheus Data Source
30 min · advanced
In progress
Objectives
  • Write provisioning YAML for a datasource and a dashboard provider
  • Run Prometheus and node-exporter for live data
  • Provision a dashboard as JSON with a PromQL stat panel
  • Verify provisioning through the Grafana API

Provision a Grafana Dashboard with a Prometheus Data Source

30 min · advanced

0/8 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/3
1

Which feature lets one dashboard serve many services?

2

Which Grafana feature overlays events such as deploys on a time series?

3

A dashboard should be designed to answer a specific question or drive a decision.

Quick reference

Cheat sheet

Panels and queries
time series panelMain panel for metrics over time
stat panelSingle number with threshold coloring
histogram_quantile(0.99, ...)p99 latency in a stat or graph
logs panelLive log stream inside a dashboard
$service variableTemplate variable per service
Operate Grafana
GET /api/search?type=dash-dbList dashboards via the API
provisioning/dashboards/Ship dashboards from git as JSON
datasource: prometheusPoint panels at the metrics backend
annotationsDeploy and incident markers on time
grafana-cli pluginsInstall extra visualizations
Go deeper

Official documentation

Grafana — Dashboards
docs
Grafana — Panels and Visualizations
docs
Grafana — Prometheus Data Source
docs
Grafana — Provisioning
docs