Neural Networks
Intermediate 15 min read Neural Networks

Gradient Descent for Neural Networks

Walking the loss surface, one step at a time

Gradient descent walks parameters downhill along the negative of the loss gradient. Batch, mini-batch and stochastic variants trade gradient noise against compute and convergence behavior.

gradient descentSGDmini-batchlearning rateconvergencelocal minima
Prerequisites: Forward Propagation
Before we start

What you will learn

  • Explain the update rule and bias-variance of the gradient estimate
  • Compare full, mini-batch and stochastic descent
  • Diagnose divergence and vanishing update patterns
Introduction

What is this?

Gradient descent minimizes a loss by repeatedly moving each parameter in the direction that most decreases it: w ← w − lr·∇L. Batch GD uses the whole dataset per step (clean, expensive), SGD uses one sample (noisy, cheap), and mini-batch sits between — noise regularizes, batching parallelizes. The learning rate sets how far each move is; too big diverges, too small creeps.

Motivation

Why do you need it?

  • It is the engine behind every scale of training from tiny MLPs to LLMs.
  • Mini-batch noise helps escape razor-thin local valleys and flatter optima.
  • Understanding step-size physics explains most failed trainings before you stare at architecture.
  • Modern optimizers (Adam, momentum) are descendents that fix specific gradient-descent pains.
Think about it

Mental model

Imagine

A ball rolling down a fogged landscape

Think of

Gradient descent on the loss

The ball feels only the local slope, not the map. Each bump (batch) gives an imperfect reading of the slope; taken with the right step size the ball still rolls to the valley — too big a step and it bounces out, too small and it crawls forever.

Mechanics

How it works

  • Compute the gradient over a mini-batch (chunk of N samples): g = ∇L_batch(w).
  • Bias-variance: bigger batches → less noisy g; small batches → helpful regularization noise, more steps/sec.
  • Update: w ← w − lr·g. Repeat for many batches (an 'epoch' is one pass over all data).
  • Memory scales with batch: you must hold the batch's activations for backprop.
  • Convergence to a local optimum (flat-ish region) rather than a guaranteed global one — iterate on init/schedule.

The descent loop.

Mini-batch
Forward
Backward
Update
Next batch
Real world

In production

  • Every SGD/Adam/AdamW training run in modern frameworks is gradient descent with bells on.
  • Batch size trade-offs are tuned (small for generalization, large for throughput and batch-normalization statistics).
  • Distributed training averages gradients across workers — the same update rule, communicated better.
Guidelines

Best practices

  • Start with a moderate batch (32-256) unless you have a reason to go bigger or smaller.
  • Normalize features and center targets so gradients aren't dominated by scale.
  • Watch normalized gradient norms; they should trend down smoothly.
  • Decay LR or use adaptive optimizers to keep the final move size sane.
Watch out

Common mistakes

  • Setting LR from 'experience' instead of an LR range test — the #1 cause of stuck or diverging runs.
  • Using the whole dataset to compute a clean gradient and paying accuracy for batch noise.
  • Forgetting that batch size changes the effective LR at Adam-level realism (scale LR with sqrt(batch)).
  • Judging convergence by train loss in non-vanishing ranges; val decides.
Root cause analysis

Common issues

Error
Loss oscillates without improving
Root cause

LR too large relative to the current gradient slope

Fix

Lower LR or add cosine/step decay after warm steps.

Error
Loss decreases for 5 epochs then explodes
Root cause

Late-training weights reached a region where LR is now too big (or BN stats degraded)

Fix

Add a LR schedule (decay) and gradient clipping; re-evaluate batch statistics.

Error
Both train and validation barely move
Root cause

LR microscopic or gradients vanishing through saturating activations

Fix

Raise LR by 10x and confirm gradient norms grow; check activation choice.

When things break

Debugging tips

  • If loss ascends immediately, drop LR 10x — almost always LR, not the model.
  • If loss reduces then bounces, reduce LR late in training or add LR decay.
  • Compare a 1-batch run with a 32-batch run; agreement in magnitude suggests healthy gradients.
  • Print grad-norm per layer once; sudden 10x jumps point to instability upstream.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

What does the update rule w ← w − lr·g represent?

2

Smaller mini-batches give cheaper gradient estimates but add noise.

Go deeper

Official documentation

An overview of gradient descent algorithms
docs