GKE OOM events: container-level vs node-level kills
An OOM kill is container-level (limit reached) or system-level (node out of memory). Confirm with OOMKilled and exit 137, and search logs for hidden ones.
What you see
Reason: OOMKilled,Exit Code: 137- Containers restart with no obvious error
- Pods vanish from a node under memory pressure
What it means
The Linux OOM killer handles every out-of-memory event. A container-level kill happens when a container tries to exceed its memory limit: the kernel first tries to reclaim memory in the container's cgroup, then kills a process and Kubernetes marks the container OOMKilled.
A system-level kill happens when the node itself runs out of memory, regardless of a container's own limit. Google's guide also covers 'invisible' OOM events, where a process inside the container is killed but the container itself keeps running.
How to confirm the cause
- 1Check for a visible OOM
kubectl describe pod POD_NAME
Reason: OOMKilledandExit Code: 137underLast Stateconfirm it. An exit code of 137 also indicates an OOM kill. - 2Search logs for invisible events
# Logs Explorer: search the node and container logs for OOM messages around the restart time
Google's guide has this as its own step.
- 3Watch node logs live
journalctl -k -f | grep -i oom
Run on the node. Use
-kfor kernel messages and-fto follow.
Causes and fixes
Memory limit too low
- HOW TO TELL
- The container is killed soon after reaching its normal working set.
- ROOT CAUSE
resources.limits.memoryis below the app's typical or peak demand.- FIX
- Raise the limit (and the request) to cover measured peak usage.
No requests or limits set
- HOW TO TELL
- Neither
requests.memorynorlimits.memoryis defined. - ROOT CAUSE
- Memory use is unbounded, so one pod can exhaust the node and trigger system-level kills.
- FIX
- Set both requests and limits on every container.
A spike in load
- HOW TO TELL
- Kills line up with traffic peaks.
- ROOT CAUSE
- Sudden surges push memory past the limit even when it is usually adequate.
- FIX
- Raise the limit or scale out earlier, and bound caches and queues.
How to stop it coming back
- Set memory requests and limits, based on measured peaks.
- Alert on restarts and on memory use near the limit.
- Use pod autoscaling for load spikes.
Keep going
- Runbook: GKE CrashLoopBackOff: read the exit code, then the liveness probe
- Runbook: OOMKilled and exit code 137
- Runbook: Process killed by the out-of-memory killer
- Cheat sheet: kubectl
- Cheat sheet: journalctl
- Lesson: the kubernetes topic in Learn
Sources
Google Cloud documentation (docs.cloud.google.com), read 18 Nov 2026.