Google CloudSenior

GKE OOM events: container-level vs node-level kills

An OOM kill is container-level (limit reached) or system-level (node out of memory). Confirm with OOMKilled and exit 137, and search logs for hidden ones.

What you see

  • Reason: OOMKilled, Exit Code: 137
  • Containers restart with no obvious error
  • Pods vanish from a node under memory pressure

What it means

The Linux OOM killer handles every out-of-memory event. A container-level kill happens when a container tries to exceed its memory limit: the kernel first tries to reclaim memory in the container's cgroup, then kills a process and Kubernetes marks the container OOMKilled.

A system-level kill happens when the node itself runs out of memory, regardless of a container's own limit. Google's guide also covers 'invisible' OOM events, where a process inside the container is killed but the container itself keeps running.

How to confirm the cause

  1. 1
    Check for a visible OOM
    kubectl describe pod POD_NAME

    Reason: OOMKilled and Exit Code: 137 under Last State confirm it. An exit code of 137 also indicates an OOM kill.

  2. 2
    Search logs for invisible events
    # Logs Explorer: search the node and container logs for OOM messages around the restart time

    Google's guide has this as its own step.

  3. 3
    Watch node logs live
    journalctl -k -f | grep -i oom

    Run on the node. Use -k for kernel messages and -f to follow.

Causes and fixes

  • Memory limit too low

    HOW TO TELL
    The container is killed soon after reaching its normal working set.
    ROOT CAUSE
    resources.limits.memory is below the app's typical or peak demand.
    FIX
    Raise the limit (and the request) to cover measured peak usage.
  • No requests or limits set

    HOW TO TELL
    Neither requests.memory nor limits.memory is defined.
    ROOT CAUSE
    Memory use is unbounded, so one pod can exhaust the node and trigger system-level kills.
    FIX
    Set both requests and limits on every container.
  • A spike in load

    HOW TO TELL
    Kills line up with traffic peaks.
    ROOT CAUSE
    Sudden surges push memory past the limit even when it is usually adequate.
    FIX
    Raise the limit or scale out earlier, and bound caches and queues.

How to stop it coming back

  • Set memory requests and limits, based on measured peaks.
  • Alert on restarts and on memory use near the limit.
  • Use pod autoscaling for load spikes.

Keep going

Sources

Google Cloud documentation (docs.cloud.google.com), read 18 Nov 2026.

Search

Search cheat sheets and projects