criticalIntermediateUSE#oom#cgroup#memory#dmesg#limits

Cgroup OOM kill — process killed by kernel out-of-memory

dmesg shows Out of memory: Killed process <PID>

Likely source:availabilityperformance

Focused incident mode — strips away everything but this runbook.

Symptoms

  • dmesg shows Out of memory: Killed process <PID>
  • Process disappears unexpectedly under memory load
  • systemd reports the service as failed with exit code 137 (SIGKILL)

OOM kills are by design — the kernel protects the system from runaway processes. The fix is either raising the limit or reducing the workload, not disabling the OOM killer.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • OOM kill messages

    dmesg | grep -i 'out of memory\|oom-kill\|Killed process'

  • Cgroup memory limit

    cat /sys/fs/cgroup/memory/memory.limit_in_bytes

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • OOM dmesg entry shows RSS exceeding the cgroup limit
  • Process was working fine until it reached the limit

Confirm before fixing

cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>/dev/null
READ

Check the hard limit

Fix

sysctl -w vm.memory.limit_in_bytes=<new-limit>
CHANGE

Raise the cgroup memory limit (or update the systemd unit / container spec)

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
dmesg | grep -i 'out of memory\|Killed process' | tail -10
READ

Find the OOM kill event

2
cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>/dev/null
READ

Check the cgroup memory limit

3
ps aux --sort=-%mem | head -10
READ

Identify current top memory consumers

4
systemctl restart <service>
CHANGE

Restart the killed service

Prevention

  • Set memory limits with a 30-40% buffer above peak RSS.
  • Alert on memory usage exceeding 80% of the cgroup limit.
  • Profile the application for memory leaks before they trigger OOM.
  • Enable swap as a safety net for unexpected spikes (not as a primary memory source).

Related runbooks

Same stack, same failure family — worth a skim.

OOMKilled containers — memory limit exceededcritical

Exit code 137 on the container

KubernetesFoundationUSE3 causes
Open runbook
Memory pressure — kswapd high, swap active, system thrashinghigh

System is sluggish and unresponsive under load

LinuxIntermediateUSE2 causes
Open runbook
Cannot allocate memory — malloc fails despite free RAMhigh

Application fails with Cannot allocate memory

LinuxAdvancedUSE3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.