Big Data
Advanced 17 min read Big Data

Spark Optimization

The four levers: data, shuffle, memory, plans

Optimizing Spark is about reducing what travels and what recomputes: minimize shuffle, avoid spills, use broadcast joins, cache deliberately, and let auto-tuning (AQE) do the mechanical parts.

SparktuningAQEbroadcast joinshufflecacheskewspill
Prerequisites: Spark Internals
Before we start

What you will learn

  • Diagnose a slow Spark job from stage metrics
  • Apply broadcast joins, partition tuning and caching deliberately
  • Use Adaptive Query Execution to fix skew and shuffle automatically
Introduction

What is this?

Spark optimization is the practice of finding and removing the job's bottleneck: too much shuffle, memory pressure that spills to disk, a skew that strands most of the cluster, or a plan that reads more than needed. Standard fixes are projection/filter pushdown, broadcast small tables, right-sized partitions, deliberate caching and spill-aware memory, perfected by AQE for skew and shuffle auto-tuning.

Motivation

Why do you need it?

  • Optimization decisions routinely have 10-100x impact on job cost and latency.
  • Understanding which knob moves each bottleneck prevents cargo-cult tuning (more RAM ≠ faster).
  • Cluster cost tracks cluster utilization, so optimized jobs lower the bill.
  • Directing optimization at the plan lets you keep data small instead of adding raw compute.
Think about it

Mental model

Imagine

Reducing what you ship internationally

Think of

Spark optimization

Free shipping (broadcast join) if the package is small; choose lighter packaging (projection); consolidate to fewer trips (fewer, bigger partitions); and track customs delays (spill, skew) to fix the specific bottleneck.

Mechanics

How it works

  • Projection and filter pushdown shrink what is read (PartitionFilters, PushedFilters).
  • Broadcast joins copy a small table to every executor, avoiding a full shuffle join.
  • Partition tuning matches shuffle partition count to data size and cores.
  • Memory/spill management keeps the working set in memory; storage levels control cache behaviors.
  • AQE (spark.sql.adaptive.enabled) optimizes at runtime: coalesces post-shuffle partitions, switches join strategies from stats, and splits skewed partitions.

Optimization levers in a query plan.

Scan
Join
Shuffle
Aggregate
Write
Real world

In production

  • A batch pipeline broadcasting a 1MB country table turns a shuffle join into a no-shuffle broadcast, cutting runtime 4x.
  • AQE coalesced 200 idle shuffle partitions down to 8 for a small job, slashing overhead.
  • Salting a hot 'NULL' product key split a skewed reducer into many, removing the 10x straggler.
Guidelines

Best practices

  • Start with the plan, not the hardware: profile stages, find the shuffle/spill, then tune.
  • Prefer the two CLI-level high-value knobs first: broadcast join threshold and shuffle partitions.
  • Cache only what's reused; persist disk for very large intermediates.
  • Let AQE handle the mechanical: enable spark.sql.adaptive.enabled=true.
Watch out

Common mistakes

  • Adding executors/memory before diagnosing (skew isn't fixed by RAM).
  • Caching everything, bloating memory to force spills.
  • Treating 200 shuffle partitions as a universal default.
  • Forgetting the write-side: unruly file sizes/partitions dominate output latency.
Root cause analysis

Common issues

Error
One stage dominates the run
Root cause

The bottleneck stage is shuffle-heavy or adding data rather than reducing

Fix

Push filters earlier, broadcast the small side, and reduce the row width.

Error
Spill to disk appears
Root cause

Data too wide/heavy for the executor memory pool

Fix

Project columns earlier, lower partition row-count, or raise executor memory.

Error
AQE not helping despite enabling
Root cause

The wrong mode: AQE needs broadcastJoinSize and coalescing set; skew needs stage separation

Fix

Enable spark.sql.adaptive.coalescePartitions + focused skew tuning (spark.sql.adaptive.skewJoin).

When things break

Debugging tips

  • Time the slow stage first; read input/size vs shuffle-write.
  • A spill in the UI means memory pressure — reduce chunk or raise memory.
  • Max vs median task duration exposes skew at a glance.
  • Diff the physical plan between a slow and fast run: strategies shift silently.
Interviews

Interview checkpoint

Practice

Hands-on lab

Make a Skewed Spark Job Fast
35 min · advanced
In progress
Objectives
  • Create a dataset with a hot key to reproduce skew
  • Measure the unequal stage duration
  • Apply salting and observe the straggler disappear

Make a Skewed Spark Job Fast

35 min · advanced

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The strongest single sign of data skew in a stage is...

2

More executors always make a Spark job faster.

Go deeper

Official documentation

Tuning Spark
docs
Adaptive Query Execution
docs