Spark Internals
From the DAG to actual JVM memory
Inside Spark: how stages are planned from the DAG, how tasks and partitions map to executors, how shuffle files and caching work, and the exact roles of Catalyst and Tungsten in fast SQL execution.
What you will learn
- Trace a query from the DAG through stages and tasks
- Explain Catalyst optimization and Tungsten execution
- Describe how shuffle and cache use executor memory
What is this?
Spark internals are the machinery that turns a logical plan into fast parallel execution. Catalyst rewrites and optimizes the query, Tungsten generates compiled code and off-heap-ish execution, and the scheduler cuts the optimized plan into stages of tasks that executors run against partitions. Caching and shuffle both live in executor blocks managed by a memory manager with on-heap, off-heap and disk tiers.
Why do you need it?
- Performance debugging is impossible without understanding stages, shuffle and memory tiers.
- Catalyst/Tungsten explain the 'SQL is fast, UDFs are slow' phenomenon you meet in production.
- Knowing how partitions map to tasks lets you tune parallelism precisely.
- Interview depth around Spark typically targets exactly these internals.
Mental model
A race team preparing a car
Spark internals
The race plan (logical plan) is refined by engineers (Catalyst) who cut weight (predicate pushdown, pruning) and fit a tuned engine (Tungsten codegen); then the track crew maps each lap to a driver (stages to tasks), and the car (executor) carries fuel, tires and telemetry (cache, shuffle, memory).
How it works
- Your query becomes a logical plan; Catalyst applies rule-based optimizations (filter pushdown, constant folding, column pruning).
- The physical planner chooses join/aggregation strategies (sort-merge, broadcast, hash) and produces a physical plan.
- The scheduler cuts the physical plan at shuffle boundaries into stages.
- Each stage is split into tasks (one per partition); the cluster manager places them on executors.
- Tungsten emits optimized bytecode for the hottest operations and manages memory in binary form; shuffle writes spill files that the reducer fetches.
Spark query internals path.
In production
- A Spark SQL heavy query ships as a few wide but efficient stages because Catalyst pushed filters into Parquet and Pruned columns.
- A 100GB join chooses a broadcast join because one side is under the threshold — no shuffle at all.
- Shuffle spilled to disk under memory pressure kept a job alive where a naive setting would have OOM'd.
Best practices
- Prefer DataFrame/SQL APIs so Catalyst can optimize; UDFs force custom serialization and block codegen.
- Tune spark.sql.shuffle.partitions to your data; the default 200 was tuned for a specific era of defaults.
- Broadcast small dimension tables explicitly to avoid shuffle joins.
- Watch for spill in the UI — spill to disk is a signal to raise memory or shrink partitions.
Common mistakes
- Writing Python UDFs for operations Catalyst could do natively — 10-100x slowdown.
- Over-caching: caching everything thrashes memory and forces spills.
- Ignoring the difference between `cache()` and `persist(STORAGE_LEVEL)` for shuffled data.
- Believing more executors always means more speed (skew, driver and file I/O often throttle).
Common issues
Task is 100x slower than the medianA skewed partition — one partition holds most of the data for a sk whose key is hot
Salting hot keys, range partitioning, or adaptive skew handling (AQE).
Memory-mostly job OOMs despite cluster having plenty of RAMPer-executor overhead + poor partition memory; grid pages blown up by wide rows
Reduce per-partition rows, enable dynamic partitions/scaling, or spill.
Same SQL is fast one day, slow the nextSpark chose different strategies based on statistics (adaptive execution) or cache state
Pin caching/persistence and check the plan diffs between runs.
Debugging tips
- Read the physical plan: look for Broadcast, SortMergeJoin, ShuffleExchange and FileScan to spot costs.
- The Waterfall of StorageLevel in the UI shows what was cached vs spilled.
- Compare Input-size vs Shuffle-write to detect data blowup between stages.
- Use SQL metrics (tasks, bytes-read, spill) in the DAG UI per stage.
Interview checkpoint
Hands-on lab
- Generate and read explain plans for a real query
- Identify shuffle (Exchange), join strategy, and filter pushdown
- Compare a UDF-free query times with a UDF version
Read Spark Plans Like an Engineer
35 min · advanced
Quick quiz
Why is a SQL DataFrame expression usually much faster than a Python UDF doing the same math?
Executors store both cached data and shuffle files in memory managed by Spark's memory manager.