criticalFoundationUSE#memory#limits#oomkilled#cgroup

OOMKilled container — memory limit exceeded

docker ps shows container Exited (137) or Exited (139)

Likely source:availabilityperformance

Focused incident mode — strips away everything but this runbook.

Symptoms

  • docker ps shows container Exited (137) or Exited (139)
  • dmesg on the host shows 'Out of memory: Killed process'
  • Container restarts intermittently under load with no application error

Exit code 137 (128 + 9/SIGKILL) is almost always the kernel OOM killer. Docker marks the container state explicitly as OOMKilled=true.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • OOM event in host kernel log

    dmesg -T | grep -i 'oom\|killed process' | tail -10

  • Container memory limit

    docker inspect <CONTAINER> --format='{{.HostConfig.Memory}}'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • OOMKilled=true on the container
  • docker stats shows memory usage near the limit before crash
  • dmesg shows Killed process with total-vm near the limit

Confirm before fixing

docker stats <CONTAINER> --no-stream --format 'table {{.Name}} {{.MemUsage}} {{.MemPerc}}'
READ

Measure memory usage before OOM

Fix

docker update --memory=<NEW_LIMIT> --memory-swap=<NEW_LIMIT> <CONTAINER>
CHANGE

Raise the memory limit on the existing container

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
docker inspect <CONTAINER> --format='OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}}'
READ

Confirm OOM killed the container

2
dmesg -T | grep -i 'oom\|killed process' | tail -5
READ

Find the kernel OOM event

3
docker stats <CONTAINER> --no-stream
READ

Measure memory usage against the limit

4
docker update --memory=<NEW_LIMIT> --memory-swap=<NEW_LIMIT> <CONTAINER>
CHANGE

Raise the limit if the workload needs it

Prevention

  • Set --memory and --memory-swap on every docker run in production.
  • Monitor container memory with docker stats and alert at 80% of limit.
  • Profile the application to understand its baseline memory footprint.
  • Use --oom-kill-disable only for critical one-off tasks, never for long-running services.

Related runbooks

Same stack, same failure family — worth a skim.

Container exits immediately after start — exit code nonzerohigh

docker ps shows container Exited (1) seconds ago

DockerFoundation3 causes
Open runbook
OOMKilled containers — memory limit exceededcritical

Exit code 137 on the container

KubernetesFoundationUSE3 causes
Open runbook
High CPU throttling — containers starved by CPU limitsmedium

High response latency with low CPU saturation on the node

KubernetesAdvancedUSE2 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.