Distributed Training
Scale compute, accept coordination cost
Distributed training splits work across devices and nodes — data-parallel by default, model-parallel for giants. Your wall-clock gains, losses, and failure semantics all change the moment you scale out.
What you will learn
- Distinguish data, model and pipeline parallelism
- Explain gradient synchronization and its bottleneck
- Design failure handling for multi-node training
What is this?
Distributed training runs a model across multiple devices/nodes. Data parallelism replicates the model across workers, each on different data shards, synchronizing gradients; model/pipeline parallelism splits the model across devices. The win is wall-clock time — at the cost of coordination overhead, complexity and new failure modes.
Why do you need it?
- Some datasets/models simply take too long on one GPU.
- Throughput scales roughly linearly until synchronization bounds it.
- Large models don't fit one device — memory forces distribution.
- Mature tooling (torchrun, Horovod, DeepSpeed) makes it approachable.
Mental model
Many chefs, one kitchen
Data-parallel training
An army of chefs speeds up prep only until the dishwasher (gradient sync) becomes the bottleneck. Adding chefs helps, but coordination dishes (allreduce) must be sized too or the kitchen just fills with people.
How it works
- Data parallelism: each worker trains on a shard; periodic gradient allreduce merges updates.
- Model parallelism: layers/weights partition across devices; activations pipe through.
- Pipeline parallelism overlaps micro-batches across model shards.
- Synchronization frequency/strategy (sync vs async, gradient accumulation) trades speed for stability.
- Checkpointing and restart-on-failure are orchestrated at the job level.
Data-parallel training with gradient synchronization.
In production
- A recommendation model trains 4× faster on 8 GPUs with gradient accumulation reducing sync frequency.
- A 70B-parameter LLM needs model-pipeline parallelism because no single GPU holds it.
- A training job checkpoints every 10 minutes; a node failure resumes from the last checkpoint via the orchestrator.
Best practices
- Profile before scaling: is it compute-bound or sync-bound?
- Increase batch/accumulation to amortize sync overhead on many workers.
- Checkpoint frequently and test recovery from a mid-training node loss.
- Use mixed precision to cut memory and speed up with minimal quality impact.
Common mistakes
- Scaling out a small model — sync overhead makes it slower than one GPU.
- Throwing nodes at a sync-bound loop instead of tuning accumulation.
- No checkpointing, so any node failure restarts from zero.
- Ignoring the learning-rate schedule that must adapt to the larger effective batch.
Debugging tips
- Speedup < 1 with more nodes? Measure per-step time and sync wall.—it's coordination-bound.
- Nondeterministic results — distribution introduces ordering; pin seeds and document tolerance.
- OOM on some nodes — load imbalance in sharding or pipeline stages; check stage sizes.
Interview checkpoint
Hands-on lab
- Simulate workers training on shards
- Implement an allreduce merge of gradients
- Measure sync-bound vs compute-bound scaling
Simulate Data-Parallel Sync
25 min · advanced
Quick quiz
The most common reason parallel training speedup plateaus is:
Small models that fit on one GPU usually benefit the most from data parallelism.