highAdvancedUSE#prometheus#cardinality#labels#series#performance

Label cardinality explosion — series count jumps after a deployment or config change

Active series count jumps by 5-10x after a new deployment

Likely source:scalingperformance

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Active series count jumps by 5-10x after a new deployment
  • Scrape duration increases and targets start timing out
  • Prometheus memory usage spikes within hours of a release
  • Query latency degrades for dashboards using the newly high-cardinality metric

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Series count before and after

    curl -s 'http://<PROM-SERVER>:9090/api/v1/query?query=prometheus_tsdb_head_series' | jq '.data.result[0].value[1]'

  • Metric label value count

    curl -s 'http://<PROM-SERVER>:9090/api/v1/query?query=count by (__name__)({__name__=~"<metric>.*"})' | jq '.data.result'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • A new label (request_id, trace_id, user_id) was added to a metric in the latest release
  • Distinct values of that label grow linearly with traffic volume
  • Each unique label value combination creates a new time series

Confirm before fixing

curl -s 'http://<PROM-SERVER>:9090/api/v1/status/tsdb' | jq '.data.seriesCountByMetricName[:5]'
READ

Check which metrics have the highest series count

Fix

kubectl edit cm -n monitoring prometheus-config
CHANGE

Add metric_relabel_configs to drop the offending label: { source_labels: [request_id], regex: '.*', action: label_drop }

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
curl -s 'http://<PROM-SERVER>:9090/api/v1/status/tsdb' | jq '.data.seriesCountByMetricName[:10]'
READ

Identify the highest-cardinality metrics

2
curl -s 'http://<PROM-SERVER>:9090/api/v1/query?query=prometheus_tsdb_head_series' | jq '.data.result[0].value[1]'
READ

Record the current series count for comparison

3
kubectl edit cm -n monitoring prometheus-config
CHANGE

Add metric_relabel_configs to drop or hash the high-cardinality label

4
curl -s -X POST http://<PROM-SERVER>:9090/-/reload
CHANGE

Reload Prometheus config after the fix

Expected: HTTP 200 — configuration reloaded

Prevention

  • Enforce a label review in the observability RFC: any new label must justify its cardinality cost.
  • Use metric_relabel_configs with label_drop or label_hash for inherently high-cardinality values.
  • Set a hard series budget (e.g., max 2M) and alert at 80%.
  • Run promtool check metrics in CI to catch label cardinality before deploy.

Related runbooks

Same stack, same failure family — worth a skim.

Prometheus OOM killed — TSDB memory exhaustion from too many seriescritical

Prometheus pod gets OOMKilled (exit code 137) repeatedly

PrometheusAdvancedUSE3 causes
Open runbook
Prometheus rules file fails config reload with parse errorhigh

Prometheus config reload fails with 'error parsing YAML' or 'parse error'

PrometheusIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.