mediumIntermediateSymptom-first#loki#grafana#logs#label#matcher

Logs land in Loki but Grafana panel shows nothing — label matcher mismatch

Grafana Loki panel shows 'No logs found' but the Loki API returns data

Likely source:configuration

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Grafana Loki panel shows 'No logs found' but the Loki API returns data
  • Logs appear in Loki CLI but not in the Grafana dashboard panel
  • LogQL query works in Loki's /loki/api/v1/query_range but Grafana shows empty
  • Panel shows logs for some label values but not others

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Loki API direct query

    curl -s -G http://<LOKI-SERVER>:3100/loki/api/v1/query_range --data-urlencode 'query={namespace="<NS>"}' --data-urlencode 'limit=5' | jq '.data.result | length'

  • Grafana panel query

    In Grafana UI: Panel → Inspect → Query → copy the exact LogQL and time range

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Grafana query uses namespace="production" but Loki labels show ns="prod"
  • The Loki exporter was configured to use different label names than the Grafana dashboard expects
  • Grafana panel query works after changing namespace to ns

Confirm before fixing

curl -s 'http://<LOKI-SERVER>:3100/loki/api/v1/labels' | jq '.data'
READ

List all labels available in Loki

Fix

In Grafana UI: Edit panel → Query → change the label matcher to match the actual Loki label names
READ

Update the LogQL in the Grafana panel to use the correct label names

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
curl -s -G http://<LOKI-SERVER>:3100/loki/api/v1/query_range --data-urlencode 'query={namespace="<NS>",job="<JOB>"}' --data-urlencode 'limit=5' | jq '.data.result | length'
READ

Query Loki directly to confirm data exists

2
curl -s 'http://<LOKI-SERVER>:3100/loki/api/v1/labels' | jq '.data'
READ

List all labels in Loki

3
curl -s 'http://<LOKI-SERVER>:3100/loki/api/v1/label/namespace/values' | jq '.data'
READ

List actual values for the namespace label

4
In Grafana UI: Edit panel → Query → update label name/value to match Loki
READ

Correct the Grafana panel query

Prevention

  • Use Grafana variables ($namespace, $job) in LogQL queries instead of hardcoded label values.
  • Document the canonical label names exported by each Promtail/Fluentbit pipeline.
  • Add a Loki label values check panel at the top of log dashboards.
  • Sync Grafana dashboard labels with the actual Loki exporter config in CI.

Related runbooks

Same stack, same failure family — worth a skim.

Grafana dashboard panels show 'No data' — query returns empty resultsmedium

Dashboard panels display 'No data' or an empty chart

GrafanaFoundation3 causes
Open runbook
Loki returns 429 — ingester rate limit or query entry limit exceededhigh

Loki returns HTTP 429 with 'maximum entries limit per query exceeded'

LokiIntermediateUSE3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.