Lesson 5 of 8 in Observability and incident response

Dashboards with Grafana: charts that answer questions

Core10 min

Grafana is a web application that queries data sources, such as Prometheus, and draws the results as panels on a dashboard. It stores no metrics itself. A dashboard is a saved arrangement of panels, each one a query plus a way of drawing it.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Metrics with Prometheus: scraping and PromQL

The problem this solves

PromQL answers one question at a time in a browser tab. During an incident you want the same few answers laid out together so you can see at a glance whether traffic, errors and speed look normal.

A dashboard is useful when it is built around questions, not around every metric you have. A wall of fifty panels hides the one that matters.

A good starting set is the four golden signals described in Google's SRE book: latency, traffic, errors and saturation. Together they describe how a service looks to its users and how full it is.

How it works

Grafana asks Prometheus and draws the answer
Grafana asks Prometheus and draws the answerYou open a dashboard, Grafana runs each panel's PromQL query against Prometheus, then draws the results as graphs for you to read.1 open2 run PromQL3 draw results4 you readYouopen the dashboardGrafanaruns panel queriesPrometheusholds the dataPanelsgraphs and numbers
  • People and clients
  • Control
  • Data
  • Other

You open a dashboard, Grafana runs each panel's PromQL query against Prometheus, then draws the results as graphs for you to read.

  1. You to Grafana: 1 open
  2. Grafana to Prometheus: 2 run PromQL
  3. Grafana to Panels: 3 draw results
  4. Panels to You: 4 you read

Grafana is a front end. A data source holds the address and settings for one backend. Each panel picks a data source, runs a query against it, and chooses a visualisation such as a time series graph or a single number.

The golden signals map to PromQL like this. Traffic is the request rate. Errors are the share of requests that fail. Latency is a percentile from histogram buckets. Saturation is how full the scarcest resource is, and in the lab the queue depth stands in for it.

Read a dashboard top to bottom as a story: is traffic normal, are errors up, are requests slower, is something full. If the top row looks wrong, the rows below help you narrow down why.

Try it yourself

Grafana is already running from the Compose file. Menu names shift a little between Grafana versions, so follow the idea if a label differs.

  1. 1
    Sign in
    # open this address in your browser
    http://localhost:3000

    The default login is admin with password admin. Grafana asks you to set a new password, and you can skip that for this local lab.

  2. 2
    Add Prometheus as a data source
    # Connections > Data sources > Add data source > Prometheus
    # URL: http://prometheus:9090
    # then click Save & test

    Use the Compose service name prometheus. The address localhost would point at the Grafana container itself.

  3. 3
    Try a query in Explore
    sum(rate(app_requests_total[1m]))

    Open Explore, choose the Prometheus data source, paste the query and run it. This is the traffic signal in requests per second.

  4. 4
    Build four panels
    # Dashboards > New > New dashboard > Add visualization
    # panel "Traffic" (requests per second)
    sum(rate(app_requests_total[1m]))
    # panel "Error ratio"
    sum(rate(app_requests_total{status="error"}[1m])) / sum(rate(app_requests_total[1m]))
    # panel "Latency p95"
    histogram_quantile(0.95, sum by (le) (rate(app_request_seconds_bucket[1m])))
    # panel "Queue depth" (saturation stand-in)
    app_queue_depth

    Give each panel a clear title and a unit (for the error ratio, choose percent as a unit between 0.0 and 1.0). Save the dashboard with a name you will recognise.

What breaks, and where to look

  • Save & test fails with an error that mentions connection refused.

    WHY IT HAPPENS
    The data source URL is http://localhost:9090. Inside the Grafana container, localhost is Grafana itself.
    CHECK FIRST
    Change the URL to http://prometheus:9090 and confirm both services are on the same Compose network with docker compose ps.
  • A panel shows No data although Prometheus has the metric.

    WHY IT HAPPENS
    Common reasons are a wrong metric name, a time range that starts before the data existed, or a rate() window shorter than two scrapes.
    CHECK FIRST
    Paste the same query into Explore and into http://localhost:9090/graph. Use the metrics browser to confirm the exact name and widen the time range.
  • Your dashboard has vanished after you recreated the containers.

    WHY IT HAPPENS
    The lab Grafana container has no volume, so its saved dashboards live inside the container and are lost when it is removed.
    CHECK FIRST
    Export the dashboard as JSON from its share or export menu and keep the file. To keep data automatically, mount a volume at /var/lib/grafana.

When not to use this

  • A dashboard shows what you thought to chart. For a question nobody planned for, you still need Explore, logs or traces.
  • Do not stare at dashboards to find problems. Alerts should tell you when to look, and dashboards help you understand once you are looking.

Check yourself

Does Grafana store the metrics it draws?

No. It queries a data source such as Prometheus each time a panel loads and draws the result.

Name the four golden signals.

Latency, traffic, errors and saturation.

Why use http://prometheus:9090 instead of http://localhost:9090 as the data source URL?

Grafana runs in its own container, where localhost means itself. The Compose service name prometheus resolves to the Prometheus container.

Lesson 82 of 92 in the DevOps path

Search

Search cheat sheets and projects