highIntermediateUSE#loki#rate-limit#429#ingester#query

Loki returns 429 — ingester rate limit or query entry limit exceeded

Loki returns HTTP 429 with 'maximum entries limit per query exceeded'

Likely source:scalingperformance

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Loki returns HTTP 429 with 'maximum entries limit per query exceeded'
  • Loki returns HTTP 429 with 'ingester rate limit exceeded' on write path
  • Grafana panels show partial log results or 'rate limit' error banner
  • Promtail or Fluentbit logs show '429 Too Many Requests' on push

Loki rate limits are per-tenant by default. A single noisy namespace can exhaust the rate limit for all tenants. Always set per-tenant limits and monitor loki_ingester_streams_created_total for unusual write spikes.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Loki ingester logs

    kubectl logs -n logging -l app=loki -c ingester | grep -i '429\|rate limit\|entries limit' | tail -10

  • Loki limits config

    kubectl get cm -n logging loki-config -o yaml | grep -A10 -E 'max_entries|ingester_rate|per_stream_rate_limit'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • 429 response body contains 'maximum entries limit per query exceeded'
  • The Grafana panel has no limit set and the query returns millions of lines
  • The time range is very broad and the log volume is high

Confirm before fixing

curl -s -G http://<LOKI-SERVER>:3100/loki/api/v1/query_range --data-urlencode 'query={job="<JOB>"}' --data-urlencode 'limit=10' --data-urlencode 'start=<START-EPOCH>' --data-urlencode 'end=<END-EPOCH>' | jq '.data.result | length'
READ

Test with limit=10 to confirm the issue is entry count

Fix

kubectl edit cm -n logging loki-config
CHANGE

Increase max_entries_limit_per_query in the limits_config section

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
kubectl logs -n logging -l app=loki -c ingester | grep -i '429\|rate limit' | tail -10
READ

Identify the rate limit type

2
kubectl get cm -n logging loki-config -o yaml | grep -A15 limits_config
READ

Review current rate limits

3
curl -s -G http://<LOKI-SERVER>:3100/loki/api/v1/query_range --data-urlencode 'query={job="<JOB>"}' --data-urlencode 'limit=10' --data-urlencode 'start=<START-EPOCH>' --data-urlencode 'end=<END-EPOCH>' | jq '.data.result | length'
READ

Test with a small limit to isolate entry count vs ingestion

4
kubectl edit cm -n logging loki-config
CHANGE

Increase the relevant rate limit (per_query, ingestion, or per_stream)

Prevention

  • Set per_tenant_rate_limit and per_tenant_rate_limit_burst appropriately for each tenant.
  • Add log volume panels per tenant/namespace to detect spikes before they hit limits.
  • Use Promtail pipeline stages to drop DEBUG/TRACE logs before they reach Loki.
  • Alert on loki_ingester_chunk_entries > max_entries_limit_per_query.

Related runbooks

Same stack, same failure family — worth a skim.

Loki LogQL range query times out — context deadline exceededhigh

Loki query returns 'context deadline exceeded' or 'query timeout'

LokiIntermediateUSE3 causes
Open runbook
Logs land in Loki but Grafana panel shows nothing — label matcher mismatchmedium

Grafana Loki panel shows 'No logs found' but the Loki API returns data

LokiIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.