Distributed Computing Basics
One job, many machines, all coordinating
Distributed computing splits work across many machines using sharding, parallelism, a coordinator and fault tolerance. The hard truths are skew, shuffle, network limits and at-least-once failure semantics.
What you will learn
- Describe sharding, parallelism and the coordinator role
- Explain why network shuffle is the performance cost of distribution
- Reason about fault tolerance: what happens when a machine dies mid-job
What is this?
Distributed computing runs a single logical task across many machines. It relies on sharding (splitting data), parallelism (many workers each processing their own shard), a coordinator (tracking progress and reassigning work), and fault tolerance (re-running lost work). The promise is elasticity; the price is network shuffle, coordination, and the failure modes of anything that can crash independently.
Why do you need it?
- You can only grow storage and compute so far vertically before cost explodes.
- Distribution gives redundancy — several copies of data survive a node failure.
- Processing in parallel reduces time-to-result for embarrassingly parallel workloads.
- It's the foundation underneath Spark, cloud warehouses and object-store analytics.
Mental model
Splitting a novel among readers
Distributed computing
Give 100 readers 1% of the book each and everyone reads in parallel — but to write the summary, all 100 must mail their notes to one editor (the shuffle + driver), and if a reader disappears you must re-assign their chapter.
How it works
- Data is divided into partitions; each partition lives on one node with a copy on another.
- A driver/coordinator maps tasks to nodes, tracking success and failure.
- Each worker runs the same operation against its own partition (map step).
- When results must be combined across partitions, a shuffle ships intermediate data over the network to the nodes that need it (reduce step).
- On failure, the coordinator re-runs the failed partition's tasks; with checkpointing, only recent work restarts.
The distributed compute lifecycle.
In production
- Spark materializes intermediate results to disk between stages to survive executor loss.
- A cloud warehouse automatically adds query slots for a big scan, then scales back down.
- An object store keeps 3 replicas of each file block so a rack failure loses no data.
Best practices
- Choose a partition count that matches your cores and data size — too many tiny tasks waste scheduling, too few under-uses the cluster.
- Design to minimize shuffle: filter and aggregate before joins/group-bys.
- Make jobs restartable: idempotent writes plus checkpoints make a node death a non-event.
- Bound skew early — a single hot key can make 99 workers wait on one.
Common mistakes
- Forgetting the network: shuffling 10GB across a cluster is often slower than the original computation.
- Assuming workers are reliable — a 1% failure rate across 10,000 tasks is 100 failures.
- Choosing huge partitions that a single machine can't process in memory.
- Ignoring stragglers: the job is only as fast as its slowest task.
Common issues
One task in the stage runs 100x longer than the restA hot key or skewed partition concentrates work on one node
Salting keys, range partitioning, or splitting the hot partition fixes stragglers.
Job fails with 'connection reset' on random tasksExecutors dying under memory pressure; the coordinator re-runs but never stabilizes
Check executor heap/GC, reduce partition size, or spill to disk instead of holding in memory.
Every run takes the same time no matter how many nodes are addedA non-parallelizable stage (driver-local collect, single postgres sink) bottlenecks the job
Parallelize the tail (multiple writers) or split the job so the serial part is minimal.
Debugging tips
- Check per-stage timing: shuffle-heavy stages mean data movement, not compute, is the limit.
- Look for data skew by checking max vs median task duration.
- Watch the driver — if it OOMs, the whole job dies even though workers are fine.
- Use a small dataset first to validate results before debugging at scale.
Interview checkpoint
Hands-on lab
- Split a dataset into shards to model distributed storage
- Parallelize a word-count over shards and merge results
- Simulate a node failure and re-run only the lost shard
Simulate Sharding, Shuffle and a Node Failure
30 min · intermediate
Quick quiz
A Spark job's shuffle is suddenly huge. What is the most likely effect?
A distributed job is as fast as its slowest (straggler) task.