Backpropagation & Learning
How the model actually learns
Backpropagation is the chain rule applied to networks: compute loss, then derive each weight's gradient layer by layer, moving backward. It is why 'learning' works, and understanding it demystifies training failures.
What you will learn
- Explain gradient descent and the role of the chain rule
- Identify exploding/vanishing gradient pathologies
- Read a learning-rate failure from loss curves
What is this?
Backpropagation computes, for every weight, how much a tiny change would reduce the loss — via the chain rule propagated backwards through the network. Gradient descent then nudges weights opposite those gradients. Training failure modes (exploding or vanishing gradients, too-high or too-low learning rate) all live in this single mechanism.
Why do you need it?
- It is the exact reason modern ML trains at all — no training, no AI.
- Exploding/vanishing gradients explain most deep-training blowups you will debug.
- Learning-rate pathologies are the most common silent training failure.
- LLM fine-tuning inherits all of it: a bad LR wrecks LoRA adapters the same way.
Mental model
Descending a mountain in fog
Gradient descent
You can't see the valley, so you feel which way the ground drops sharply and step that way. Descent with a too-big stride overshoots the slope; with a too-small stride you camp on a shelf forever. Backprop is that sense of direction; learning rate is your stride.
How it works
- The loss summarizes how wrong predictions are on a batch.
- The chain rule flows error derivatives from output back through each layer's weights.
- An optimizer (SGD, Adam) updates weights using gradient + learning-rate + momentum.
- Repeated batches/epochs lower loss; validation loss tracks generalization.
- Monitoring gradient norms exposes vanished or exploded flows early.
Forward, then backward, then step.
In production
- Fine-tuning a small adapter with AdamW and a low LR — the success of PEFT/LoRA depends on gradient stability.
- Layer-freezing (training only the head) relies on gradients stopping at the frozen boundary.
- Gradient-norm monitoring in training dashboards catches instability before loss looks bad.
Best practices
- Track gradient norms on every training run.
- Warm up the learning rate at the start (LR warm-up).
- Pretesting Adam with LR ~1e-4..3e-4 is a sane default for fine-tuning.
- Validate that gradients actually change parameters (a dry-run one batch).
Common mistakes
- Blindly trusting 'it trained' without looking at loss AND gradient curves.
- Fine-tuning a huge model with LR meant for full training — guarantees instability.
- Reading a flat loss as 'converged' when it is a stuck shelf from too-small LR.
Common issues
Loss goes to NaN mid-trainingExploding gradients through unstable layers
Lower LR, add gradient clipping, improve normalization/init.
Training never moves the modelVanishing gradients or frozen-feature wiring
Check gradient norms, unfreeze/pass gradients, use modern activations.
Validation loss rises after few stepsLearning rate too high for the data/scale
Reduce LR, shorten warm-up, and watch a LR-vs-loss sweep.
Debugging tips
- Loss unchanged but gradients near zero → vanishing flow or plateau.
- Loss nan becomes gradient norm exploding — clamp/renormalize or lower LR.
- Validation diverges while train improves → LR too high or batch too small.
Interview checkpoint
Quick quiz
What does the learning rate control?
Gradient norms should be monitored during training.