Big Data
Advanced 17 min read Big Data

Data Skew and Partitioning

The problem of the heavy partition

Skew is the silent killer of parallel jobs: a few hot keys or partitions do all the work while the cluster idles. Partitioning done right (by the natural query dimension) prevents skew, and salting/AQE fix it when it appears.

skewpartitioninghot keysaltingstragglerrepartitionbucketing
Prerequisites: Spark Internals
Before we start

What you will learn

  • Recognize skew from timing and partition-size data
  • Prevent skew with deliberate partitioning and bucketing
  • Fix hot keys with salting and adaptive execution
Introduction

What is this?

Data skew means some partitions (or keys) hold far more data than others, so a few tasks run an order of magnitude longer and the cluster idles waiting for them. Distribution (repartition, bucketing) chooses how data is split; skew is what happens when that distribution is uneven — hot users, null keys, or a dominating join value.

Motivation

Why do you need it?

  • Skew wastes the majority of your cluster for the duration of one straggler.
  • Shuffle joins amplify skew: the hot key's reducer waits for gigabytes others never see.
  • Fixed by engineering, not hardware: salting, range partitioning, or AQE.
  • Scheduling a skew fix is cheaper than the repeated cost of wasted compute.
Think about it

Mental model

Imagine

Ferry boats crossing a river

Think of

Partitioning and skew

Thirty identical boats leave at once, but one dock sends 90% of the cars — so twenty-nine boats idle while the big wait. You'd rebalance the loads across all docks (partition) or split the heavy group (salt).

Mechanics

How it works

  • Partitions are the units of parallelism: one task per partition per stage.
  • Hash partitioning routes each key to one partition, reliable but skew-prone (a hot key → one partition).
  • Range partitioning sorts keys into contiguous ranges, balancing by value spread.
  • Bucketing pre-partitions on a column so later joins/aggregations can skip shuffles entirely.
  • Remedies: salt hot keys (append a random suffix to split load), plus AQE's skewJoin splitting.

Skew vs balanced partitioning.

Hash partition
Straggler
Idle pool
Salt + range
Balanced
Real world

In production

  • A null customer key swallowing a join is split by salting so no single reducer babysits the NULLs.
  • Range partitioning on order_date keeps daily range scans balanced and avoids month-boundary hotspots.
  • Bucketing fact tables on customer_id makes customer joins probe-free (collocated).
Guidelines

Best practices

  • Pick the partition dimension from your most common filter/join — usually time or tenant.
  • Keep an eye out for NULL/empty keys — they're the classic silent hot key.
  • Bucketed tables (on the join key) reduce shuffle data dramatically for repeated joins.
  • Let AQE split skewed joins; reserve manual salting for the worst offenders.
Watch out

Common mistakes

  • Partitioning on a high-cardinality-but-skewed column (like user_id of one app's power user).
  • Salting the wrong key, wrecking the second-hop aggregation semantics.
  • Ignoring a single massive partition and 'fixing' the cluster instead.
  • Bucketing with too many buckets so each bucket is tiny — a new kind of skew.
Root cause analysis

Common issues

Error
One join task alone eats 30 minutes
Root cause

A hot join key or NULL values concentrated into one partition

Fix

Salt the key (split), split the NULLs with another dimension, or run the hot-key rows through a separate path.

Error
AQE not splitting a skewed join
Root cause

Config disabled or the skew threshold not met

Fix

Enable spark.sql.adaptive.skewJoin.enabled and tune skewJoin.skewedPartitionFactor/threshold.

Error
Bucketed table not avoiding a shuffle
Root cause

Bucket count/type mismatch between fact and dimension

Fix

Match the bucket number and bucket column exactly across joined tables.

When things break

Debugging tips

  • Look at max vs median task duration in the UI — 10x gap means skew.
  • Check per-partition byte counts with sizeInBytes before joins.
  • MySQL NULL hot keys show up as a single dominant reduce task.
  • Compare logical vs physical plan to see whether AQE actually split the skew.
Interviews

Interview checkpoint

Practice

Hands-on lab

Prove Skew and Balance It with DuckDB
30 min · advanced
In progress
Objectives
  • Build a dataset with an extreme hot key
  • Demonstrate the imbalance by per-partition sizes
  • Rebalance with a salted second grouping and compare

Prove Skew and Balance It with DuckDB

30 min · advanced

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The classic silent hot key in production data is...

2

Range partitioning is symmetrical with hash partitioning for all workloads.

Go deeper

Official documentation

Spark tuning & partitioning
docs
AQE skew join options
docs