Big Data
Intermediate 16 min read Big Data

Spark Architecture

Driver, DAG, executors and lineage

Spark runs on a driver that builds a directed acyclic graph (DAG) of transformations and executes it as stages of parallel tasks on executors. In-memory caching is what makes it fast, and lineage is what makes it fault-tolerant.

SparkdriverexecutorDAGRDDstagetasklineage
Prerequisites: Distributed Computing Basics
Before we start

What you will learn

  • Describe the roles of the driver, executors and cluster manager
  • Explain how wide and narrow transformations shape the DAG into stages
  • Use lineage to reason about fault tolerance
Introduction

What is this?

Apache Spark is a distributed processing engine built on the mapreduce idea but with two upgrades: it keeps intermediate data in memory, and it records the computation as a DAG so resilient distributed datasets (RDDs) can be rebuilt from lineage after a failure. A driver runs the user program, builds the DAG, and schedules stages of tasks across executors that hold data and compute.

Motivation

Why do you need it?

  • In-memory processing makes iterative and interactive workloads dramatically faster than disk-based MapReduce.
  • The DAG optimizer reduces the number of shuffles compared to naive map-shuffle-reduce.
  • Lineage-based recomputation rebuilds lost partitions without keeping a full copy of everything.
  • Spark speaks DataFrame/SQL, making it usable by analysts as well as engineers.
Think about it

Mental model

Imagine

A restaurant with a head chef directing stations

Think of

Spark driver and executors

The head chef (driver) plans the whole meal as a recipe graph (DAG) and tells stations (executors) what to prepare. If a station drops a plate, the chef can 'redo' it from the recipe — lineage — rather than starting dinner over.

Mechanics

How it works

  • The driver runs your program, building an execution plan (the DAG) of transformations over RDDs/DataFrames.
  • Transformations are lazy: no compute happens until an action (collect, count, write) forces it.
  • Narrow transformations (filter, map on one partition) stay within a single stage; wide transformations (groupBy, join) trigger a shuffle and start a new stage.
  • The scheduler splits the DAG into stages, then into tasks, one per partition, and sends tasks to executors.
  • Executors cache data in memory for reuse; on failure, the driver recomputes lost partitions from lineage.

From user program to the DAG to stages

Driver
Lazy DAG
Shuffle boundaries
Tasks
Executors
Real world

In production

  • A streaming company runs nightly batch ETL on Spark SQL against Parquet, caching hot dimension tables.
  • A fraud team iterates an ML feature job, reusing a cached processed dataset across days of experiments.
  • Spark connects to thousands of S3 objects, executing a single logical query across the cluster.
Guidelines

Best practices

  • Design for few, wide-grained partitions (aim 1–4 tasks per core) rather than per-field task storms.
  • Cache only datasets reused across multiple actions; caching everything wastes memory and spooling.
  • Use CLI-level parallelism: set spark.sql.shuffle.partitions sensibly (not 200 for tiny and not 2000 for big).
  • Persist to disk for very large intermediate results you can't hold in memory.
Watch out

Common mistakes

  • Calling actions inside a loop, re-reading/recomputing the same data each iteration.
  • Cache with wrong storage level so memory spills to disk and performance collapses.
  • Assuming Spark has loaded data when you've only built a lazy DataFrame — it hasn't.
  • Ignoring the driver: a driver OOM kills the job even when executors are healthy.
Root cause analysis

Common issues

Error
Job fails at the driver with OOM while executors are idle
Root cause

A collect() or toPandas() pulled an entire dataset into driver memory

Fix

Aggregate/shuffle-partition results, or replace collect with streaming writes to storage.

Error
Same job finishes slower every run
Root cause

Cached data evicted and regenerated, or spilled to disk under memory pressure

Fix

Right-size executor memory, cache only reused datasets, and monitor storage levels.

Error
UI shows huge shuffle reads each stage
Root cause

Wide operations (join/groupBy) happening early instead of after filtering/aggregation

Fix

Push projection and filtering down before the shuffle to shrink the data moved.

When things break

Debugging tips

  • Compare the DAG UI stage timings: shuffle stages are where most wins live.
  • A 'detected memory leak' or driver OOM points at a collect() pulling everything to the driver.
  • Watch input-size vs shuffle-write to see whether transformations shrink or blow up the data.
  • Use spark.eventLog and the history server to debug a job after it dies.
Interviews

Interview checkpoint

Practice

Hands-on lab

Run Word Count on a Local Spark Standalone Cluster
35 min · intermediate
In progress
Objectives
  • Launch Spark in standalone mode and confirm the master/worker topology
  • Run a DataFrame word count and inspect the DAG
  • Observe lazy evaluation by timing a transformation vs an action

Run Word Count on a Local Spark Standalone Cluster

35 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which of these triggers a shuffle (new stage)?

2

Building a DataFrame reads the whole dataset immediately.

Go deeper

Official documentation

Spark cluster overview
docs
Spark programming guide (RDD/DataFrame)
docs