criticalAdvancedUSE#prometheus#tsdb#oom#memory#series

Prometheus OOM killed — TSDB memory exhaustion from too many series

Prometheus pod gets OOMKilled (exit code 137) repeatedly

Likely source:availabilityscaling

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Prometheus pod gets OOMKilled (exit code 137) repeatedly
  • process_resident_memory_bytes is near the container memory limit
  • Number of active time series grows steadily before each crash
  • TSDB head chunks are large and compaction falls behind

Prometheus memory is proportional to active series, not scrape volume. A single bad label (e.g., user_id) can 10x your series count overnight. Always monitor series cardinality as a leading indicator.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • TSDB series count

    curl -s http://<PROM-SERVER>:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[:10]'

  • Container memory usage vs limit

    kubectl top pod -n monitoring -l app=prometheus --containers

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • seriesCountByMetricName shows metrics with label values in the millions
  • Labels like user_id, request_id, or trace_id create a new series per unique value
  • Series count grows linearly with traffic, not with distinct endpoints

Confirm before fixing

curl -s http://<PROM-SERVER>:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[:5]'
READ

Identify the highest-cardinality metrics

Fix

kubectl edit cm -n monitoring prometheus-config
CHANGE

Add metric_relabel_configs to drop or aggregate high-cardinality labels before they hit TSDB

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
kubectl get pod -n monitoring -l app=prometheus -o jsonpath='{.items[0].status.containerStatuses[0].lastState.terminated.reason}'
READ

Confirm OOMKilled

2
curl -s http://<PROM-SERVER>:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[:10]'
READ

Find the highest-cardinality metrics

3
curl -s 'http://<PROM-SERVER>:9090/api/v1/query?query=count({__name__!=""}) by (job)' | jq '.data.result[:5]'
READ

Narrow down to the offending job

4
kubectl edit cm -n monitoring prometheus-config
CHANGE

Add metric_relabel_configs to drop high-cardinality labels

Prevention

  • Set process_resident_memory_bytes alert at 80% of container memory limit.
  • Use metric_relabel_configs to drop or hash high-cardinality labels before TSDB ingestion.
  • Cap the number of active series with TSDB storage.tsdb.max-head-chunks-active-series-limit.
  • Monitor series count trend daily with a recording rule: count(count by (__name__)({__name__!=""})).

Related runbooks

Same stack, same failure family — worth a skim.

OOMKilled containers — memory limit exceededcritical

Exit code 137 on the container

KubernetesFoundationUSE3 causes
Open runbook
Label cardinality explosion — series count jumps after a deployment or config changehigh

Active series count jumps by 5-10x after a new deployment

PrometheusAdvancedUSE3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.