highAdvancedSymptom-first#loki#retention#storage#index#wal

Loki logs missing before retention period — early purge or index gap

Loki query for logs older than a few days returns no results

Likely source:data-lossconfiguration

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Loki query for logs older than a few days returns no results
  • Index shows a gap — logs from certain hours/days are missing entirely
  • Loki querier logs show 'index not found' or 'chunk not found' for the affected period
  • WAL replay fails and recent logs are lost after a restart

Loki retention is enforced by the compactor, not by the ingester. If you restart the compactor with a short retention_period, it will immediately begin purging old data. Always verify retention settings before running the compactor.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Loki retention config

    kubectl get cm -n logging loki-config -o yaml | grep -A5 'retention_period\|retention\|compactor'

  • Loki storage listing

    curl -s 'http://<LOKI-SERVER>:3100/loki/api/v1/labels' | jq '.data'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • retention_period in loki-config is set to 24h but logs should be kept for 30 days
  • The compactor runs every 10 minutes and deletes chunks older than retention_period
  • Logs from 2 days ago are completely gone

Confirm before fixing

kubectl get cm -n logging loki-config -o yaml | grep retention_period
READ

Check the retention period

Expected: retention_period: 24h or similar short value

Fix

kubectl edit cm -n logging loki-config
CHANGE

Increase retention_period to the desired value (e.g., 720h for 30 days)

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
kubectl get cm -n logging loki-config -o yaml | grep -A10 -E 'retention_period|compactor|schema'
READ

Review retention and compaction config

2
kubectl logs -n logging -l app=loki -c compactor | grep -i 'flush\|compact\|delete' | tail -20
READ

Check compactor activity

3
kubectl logs -n logging -l app=loki -c querier | grep -i 'index not found\|chunk not found' | tail -10
READ

Identify the exact error for the missing period

4
kubectl edit cm -n logging loki-config
CHANGE

Fix retention_period and compactor settings

Prevention

  • Set retention_period explicitly in the limits_config and match it to your compliance requirements.
  • Run the compactor with a 24h grace period before enforcing new retention settings.
  • Monitor loki_compactor_deleted_entries_total to detect unexpected purges.
  • Use object lifecycle policies (S3/GCS) as a secondary retention enforcement layer.

Related runbooks

Same stack, same failure family — worth a skim.

Loki LogQL range query times out — context deadline exceededhigh

Loki query returns 'context deadline exceeded' or 'query timeout'

LokiIntermediateUSE3 causes
Open runbook
Logs land in Loki but Grafana panel shows nothing — label matcher mismatchmedium

Grafana Loki panel shows 'No logs found' but the Loki API returns data

LokiIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.