criticalFoundationUSE#memory#limits#oomkilled#performance

OOMKilled containers — memory limit exceeded

Exit code 137 on the container

Likely source:availabilityperformance

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Exit code 137 on the container
  • kubectl describe shows terminated.reason = OOMKilled
  • Container restarts intermittently under load with no app error

137/SIGKILL from the kernel means the cgroup limit was hit — this is expected behavior, sized wrong, not a cloud failure.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • OOM confirmation

    kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

  • Memory trend

    kubectl top pod <pod> -n <namespace> --containers

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Usage peaks near the limit during normal traffic
  • Baseline usage is stable, no leak pattern in logs

Confirm before fixing

kubectl top pod <pod> -n <namespace> --containers
READ

Observe peak vs limit

Fix

kubectl set resources deployment/<deploy> --limits=memory=<new> --requests=memory=<new>
CHANGE

Raise the limit with headroom

Expected: deployment.apps/<deploy> resource requirements updated

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
READ

Confirm OOMKilled

2
kubectl top pod <pod> -n <namespace> --containers
READ

Measure actual usage to size the limit

3
kubectl set resources deployment/<deploy> --limits=memory=<size> --requests=memory=<size>
CHANGE

Right-size the memory limit

4
kubectl rollout status deployment/<deploy> -n <namespace>
READ

Confirm the rollout completes

Prevention

  • Set limits above peak observed usage with a 30-40% buffer.
  • Deploy VPA in recommendation mode to keep limits tuned.
  • Alert on memory usage > 80% of limit (Prometheus: container_memory_working_set_bytes).
  • Use -XX:MaxRAMPercentage or equivalent for JVM/Node in containers.

Related runbooks

Same stack, same failure family — worth a skim.

CrashLoopBackOff — container keeps crashing and restartingcritical

kubectl get pods shows CrashLoopBackOff status

KubernetesFoundation4 causes
Open runbook
High CPU throttling — containers starved by CPU limitsmedium

High response latency with low CPU saturation on the node

KubernetesAdvancedUSE2 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.