Lesson 5 of 8 in Observability and incident response
Dashboards with Grafana: charts that answer questions
Grafana is a web application that queries data sources, such as Prometheus, and draws the results as panels on a dashboard. It stores no metrics itself. A dashboard is a saved arrangement of panels, each one a query plus a way of drawing it.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Metrics with Prometheus: scraping and PromQL
The problem this solves
PromQL answers one question at a time in a browser tab. During an incident you want the same few answers laid out together so you can see at a glance whether traffic, errors and speed look normal.
A dashboard is useful when it is built around questions, not around every metric you have. A wall of fifty panels hides the one that matters.
A good starting set is the four golden signals described in Google's SRE book: latency, traffic, errors and saturation. Together they describe how a service looks to its users and how full it is.
How it works
- People and clients
- Control
- Data
- Other
You open a dashboard, Grafana runs each panel's PromQL query against Prometheus, then draws the results as graphs for you to read.
- You to Grafana: 1 open
- Grafana to Prometheus: 2 run PromQL
- Grafana to Panels: 3 draw results
- Panels to You: 4 you read
Grafana is a front end. A data source holds the address and settings for one backend. Each panel picks a data source, runs a query against it, and chooses a visualisation such as a time series graph or a single number.
The golden signals map to PromQL like this. Traffic is the request rate. Errors are the share of requests that fail. Latency is a percentile from histogram buckets. Saturation is how full the scarcest resource is, and in the lab the queue depth stands in for it.
Read a dashboard top to bottom as a story: is traffic normal, are errors up, are requests slower, is something full. If the top row looks wrong, the rows below help you narrow down why.
Try it yourself
Grafana is already running from the Compose file. Menu names shift a little between Grafana versions, so follow the idea if a label differs.
- 1Sign in
# open this address in your browser http://localhost:3000
The default login is
adminwith passwordadmin. Grafana asks you to set a new password, and you can skip that for this local lab. - 2Add Prometheus as a data source
# Connections > Data sources > Add data source > Prometheus # URL: http://prometheus:9090 # then click Save & test
Use the Compose service name
prometheus. The addresslocalhostwould point at the Grafana container itself. - 3Try a query in Explore
sum(rate(app_requests_total[1m]))
Open Explore, choose the Prometheus data source, paste the query and run it. This is the traffic signal in requests per second.
- 4Build four panels
# Dashboards > New > New dashboard > Add visualization # panel "Traffic" (requests per second) sum(rate(app_requests_total[1m])) # panel "Error ratio" sum(rate(app_requests_total{status="error"}[1m])) / sum(rate(app_requests_total[1m])) # panel "Latency p95" histogram_quantile(0.95, sum by (le) (rate(app_request_seconds_bucket[1m]))) # panel "Queue depth" (saturation stand-in) app_queue_depthGive each panel a clear title and a unit (for the error ratio, choose percent as a unit between 0.0 and 1.0). Save the dashboard with a name you will recognise.
What breaks, and where to look
Save & test fails with an error that mentions connection refused.
- WHY IT HAPPENS
- The data source URL is
http://localhost:9090. Inside the Grafana container,localhostis Grafana itself. - CHECK FIRST
- Change the URL to
http://prometheus:9090and confirm both services are on the same Compose network withdocker compose ps.
A panel shows
No dataalthough Prometheus has the metric.- WHY IT HAPPENS
- Common reasons are a wrong metric name, a time range that starts before the data existed, or a
rate()window shorter than two scrapes. - CHECK FIRST
- Paste the same query into Explore and into
http://localhost:9090/graph. Use the metrics browser to confirm the exact name and widen the time range.
Your dashboard has vanished after you recreated the containers.
- WHY IT HAPPENS
- The lab Grafana container has no volume, so its saved dashboards live inside the container and are lost when it is removed.
- CHECK FIRST
- Export the dashboard as JSON from its share or export menu and keep the file. To keep data automatically, mount a volume at
/var/lib/grafana.
When not to use this
- A dashboard shows what you thought to chart. For a question nobody planned for, you still need Explore, logs or traces.
- Do not stare at dashboards to find problems. Alerts should tell you when to look, and dashboards help you understand once you are looking.
Check yourself
Does Grafana store the metrics it draws?
No. It queries a data source such as Prometheus each time a panel loads and draws the result.
Name the four golden signals.
Latency, traffic, errors and saturation.
Why use http://prometheus:9090 instead of http://localhost:9090 as the data source URL?
Grafana runs in its own container, where localhost means itself. The Compose service name prometheus resolves to the Prometheus container.
Keep going
Lesson 82 of 92 in the DevOps path