Big Data
Advanced 16 min read Big Data

Memory Management in Distributed Engines

Three tenants, one RAM budget

Distributed engines juggle three memory users at once: data caching, shuffle spill, and operator work. Understanding the memory pools and spill rules is how you stop jobs from falling off the RAM cliff.

memoryspilloff-heapOOMexecutor memorycachinggarbage collectionshuffle buffers
Prerequisites: Spark Internals
Before we start

What you will learn

  • Name the memory pools: storage, execution, overhead
  • Explain when and how spill happens
  • Diagnose OOM vs GC vs spill problems from logs
Introduction

What is this?

In distributed engines like Spark and Flink a worker's memory is split into pools: storage (for cached data), execution (for shuffles, joins and aggregations), and overhead/user (JVM/process overhead). When the execution or storage pool fills, the engine spills to disk rather than crashing; when overhead is mis-sized, the container gets killed by the scheduler.

Motivation

Why do you need it?

  • OOM kills are the #1 class of 'it works on my laptop' production failures.
  • Most 'add more RAM' fixes are misdiagnoses of a spill or caching problem.
  • Knowing the pools tells you whether to shrink partitions, change storage levels, or raise overhead.
  • Consistent memory answers show senior-level systems thinking in interviews.
Think about it

Mental model

Imagine

A small kitchen with a pantry

Think of

Executor memory + spill

Counter space (execution) is for cooking, the pantry (storage) stores prepared ingredients; when the counter fills you stack bowls in the fridge (spill to disk) rather than throw dinner away — but every trip between them slows the meal.

Mechanics

How it works

  • Storage pool holds cached RDD/DataFrame partitions (persist levels control memory vs disk).
  • Execution pool holds shuffle buffers, join hashes and aggregation state; it can evict storage when pressure makes this safe.
  • Overhead pool holds JVM/pyspark/python runtime overhead — container memory = executor + overhead.
  • On memory blowout, the engine spills partially-aggregated or shuffle data to disk and resumes.
  • Off-heap memory (Tungsten) avoids garbage-collection pauses but is separate from the on-heap pools.

How executor memory is divided.

Container RAM
Storage
Execution
Overhead
Spill
Real world

In production

  • A large groupBy oozes to disk and completes instead of OOMing because execution-spill works.
  • Caching a hot 20GB dataset with MEMORY_AND_DISK survives eviction by a later shuffle.
  • A PySpark driver raised overhead after a python-env and UDF memory spike.
Guidelines

Best practices

  • Right-size partitions so per-partition memory stays well under a single executor's execution pool.
  • Cache only reused datasets, and prefer MEMORY_AND_DISK for large-but-valuable caches.
  • Add explicit memoryOverhead for Python/container overhead (commonly 10-20%).
  • Watch the UI's storage + execution breakdown, not just the pleasant '8GB' number.
Watch out

Common mistakes

  • Setting executor memory to the node size, ensuring GC death rather than a workable job.
  • Caching everything so storage evicts execution and every query spills.
  • Misreading a spill log as a hardware problem and buying more RAM.
  • Ignoring off-heap: some operations (netty buffers) live off-heap and blow container limits.
Root cause analysis

Common issues

Error
Spark job OOMs on groupBy
Root cause

Group data exceeds the execution pool (many keys or huge aggregation state)

Fix

Enable spill, reduce partition row count and keys, or run two-phase aggregation.

Error
'container killed on memory exceeded' with no Spark exception
Root cause

Python/process overhead pushing past container RAM

Fix

Raise spark.executor.memoryOverhead and the container's hard limit.

Error
80% spill-to-disk
Root cause

Storage+cache eviction pressures execution; partitions too big

Fix

Lower cache footprint, reduce partition size, or increase executor memory.

When things break

Debugging tips

  • 'killed by YARN'/'memory limit exceeded' = container-level (overhead), usually not a code OOM.
  • Spill entries in the DAG UI point at execution pressure, not a coding bug.
  • Frequent GC pauses look like slow-but-not-crashing jobs — check GC logs.
  • Compare peak memory vs container limit in the resource manager UI.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

A 'killed by YARN container memory exceeded' error most likely means...

2

Spill to disk is always a sign the cluster is too small.

Go deeper

Official documentation

Tuning Spark (memory)
docs
Spark security/overhead configs
docs