highIntermediateSymptom-first#prometheus#rules#config#reload#alerting

Prometheus rules file fails config reload with parse error

Prometheus config reload fails with 'error parsing YAML' or 'parse error'

Likely source:configurationdeployment

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Prometheus config reload fails with 'error parsing YAML' or 'parse error'
  • Alerts and recording rules stop being evaluated after a config change
  • prometheus-config-reloader sidecar shows error logs
  • New alert rules do not fire; old rules continue to evaluate stale data

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Prometheus reload error

    kubectl logs -n monitoring deploy/prometheus -c prometheus | grep -i 'parse error\|error loading' | tail -20

  • Rule file syntax check

    promtool check rules <RULE-FILE-PATH>

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • promtool check rules outputs 'yaml: unmarshal errors' or 'bad field name'
  • Recent edit to the rule file changed indentation or added an invalid field

Confirm before fixing

promtool check rules <RULE-FILE-PATH>
READ

Validate the rule file

Expected: ERROR with specific line/column

Fix

sed -n '<LINE-NUMBER>p' <RULE-FILE-PATH>
READ

Inspect the problematic line

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
promtool check rules <RULE-FILE-PATH>
READ

Validate rule file syntax

2
kubectl logs -n monitoring deploy/prometheus -c prometheus | grep -i 'parse error\|error loading' | tail -10
READ

Check Prometheus logs for the exact error

3
curl -s -X POST http://<PROM-SERVER>:9090/-/reload
CHANGE

Reload config after fixing the rule file

4
curl -s http://<PROM-SERVER>:9090/api/v1/rules | jq '.data.groups[].rules[] | {name, health}'
READ

Verify all rules are now being evaluated (health=ok)

Prevention

  • Add promtool check rules to the CI pipeline so invalid rules never reach production.
  • Use promtool test rules with synthetic data to validate rule behavior before deploy.
  • Keep rule files under 500KB and split large rule sets across multiple ConfigMaps.
  • Set up a Prometheus reload webhook that validates syntax before triggering the reload endpoint.

Related runbooks

Same stack, same failure family — worth a skim.

Prometheus alert rule not firing — rule evaluated but alert stays inactivecritical

Alert rule shows as 'inactive' or 'pending' but never reaches 'firing' state

PrometheusIntermediateRED3 causes
Open runbook
Prometheus scrape 401/403 — metrics endpoint requires authenticationhigh

Prometheus target shows last_error containing HTTP status 401 or 403

PrometheusIntermediateUSE3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.