Deep Learning
Intermediate 16 min read Deep Learning

Backpropagation

The chain rule, applied backwards through the whole net

Backpropagation computes the gradient of the loss with respect to every weight by applying the chain rule backward through the network, enabling gradient descent to actually train deep nets.

backpropagationchain rulegradientsautogradloss gradienttraining
Prerequisites: Deep Learning Basics
Before we start

What you will learn

  • Trace how errors flow backward from the loss to each weight
  • Explain why backprop is a batch-wise local computation, not magic
  • Identify when gradients vanish or explode and what to do
Introduction

What is this?

Backpropagation is an algorithm that computes, for every weight in a network, how much the loss would change if that weight moved a little — the gradient. It works by walking the computational graph backward from the loss: at each node it applies the chain rule, multiplying the error arriving from above by the local derivative of that node. Repeat for millions of weights and you get one direction to descend.

Motivation

Why do you need it?

  • Without gradients you'd tune millions of weights blindly; backprop gives each one a precise update direction.
  • It reuses shared intermediate values (activations), so it's efficient — one forward and one backward pass per batch.
  • It's exact: the computed gradients are the true derivatives, not numerical approximations.
  • Auto-differentiation frameworks (PyTorch, TF) automate it, but understanding it is how you debug training.
Think about it

Mental model

Imagine

Blame flowing down an organization chart

Think of

Error flowing backward through a network

The CEO (loss) reports a problem. Managers don't each get blamed directly — the share each one contributed is propagated layer by layer until every individual (weight) knows its own responsibility for the result.

Mechanics

How it works

  • Forward pass: activations are computed for each layer and cached.
  • Loss: the final prediction is scored against the target.
  • Backward pass: starting from the loss, compute the gradient of the loss w.r.t. the last layer's weights (easy, local).
  • Chain rule: the gradient for an earlier layer multiplies the gradient of the layer above by its local derivative.
  • Update: each weight moves a small step opposite its gradient; the cached activations are reused so the whole thing costs about 2x a forward pass.

Gradients flow backward through the stored computational graph.

Forward
Loss
Backward
Gradients
Update
Real world

In production

  • Every neural net you train — image classifiers, LLMs, recommenders — uses backprop for its updates.
  • Transfer learning fine-tunes many layers at once using gradients computed through backprop.
  • Gradient-based hyperparameter/architecture search reuses the same machinery for meta-learning.
Guidelines

Best practices

  • Sanity-check gradients numerically or with a known one-shot case (e.g. a single neuron) before trusting custom code.
  • Monitor per-layer gradient norms during training; order-of-magnitude drift signals instability.
  • Use optimizers that leverage gradient statistics (Adam) but understand they add their own noise.
  • Enable gradient clipping near loss spikes / large batch variance.
Watch out

Common mistakes

  • Thinking backprop 'tells each neuron its error' — it computes gradient field, neurons don't know anything.
  • Recomputing rather than caching activations, wasting the 2x efficiency.
  • Chaining too many ops through a custom loss and getting NaN gradients from fp16 overflow.
  • Forgetting that depth multiplies many derivatives — see vanishing gradients.
Root cause analysis

Common issues

Error
Training loss never drops below 0.7 on a 2-class problem
Root cause

Saturated sigmoid output: gradient ~ 0 for confident-but-wrong predictions

Fix

Use logits with cross-entropy (no manual sigmoid) so the gradient stays informative.

Error
Loss goes to NaN mid-training
Root cause

Exploding gradients — weights blew up in an early batch

Fix

Gradient clip, lower LR, or better initialization; check for raw exp/log on unstable values.

Error
First layers barely change across epochs
Root cause

Vanishing gradients from many small derivatives multiplied together

Fix

Use ReLU-family activations, normalization layers, residual/skip connections and reasonable initialization.

When things break

Debugging tips

  • Print gradient norms by layer: if early layers get near-zero gradients while later are fine, gradients vanish.
  • Verify the first backward step reduces loss on a fixed, tiny batch.
  • Use one neuron/one layer to test that your manual loss gradient equals a numerical gradient.
  • Freeze layers one at a time to isolate which layer's gradients explode.
Interviews

Interview checkpoint

Practice

Hands-on lab

Implement Backprop by Hand
30 min · intermediate
In progress
Objectives
  • Implement forward and backward passes for a tiny 2-layer MLP manually
  • Derive and code the gradient of MSE loss yourself
  • Verify your manual gradients against a numerical gradient check

Implement Backprop by Hand

30 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

What mathematical rule is backpropagation built on?

2

Backpropagation reuses cached forward activations, making it roughly 2x the forward pass cost.

Go deeper

Official documentation

PyTorch autograd tutorial
docs
CS231n backprop notes
docs