RNN
Advanced 14 min read RNN

Vanishing & Exploding Gradients

Why RNNs can't remember — and what to do

BPTT multiplies many cell Jacobians; small ones erase early context gradients, large ones blow up. Gated cells, clipping and skips exist precisely because of this.

vanishing gradientexploding gradientBPTTgradient clippingspectral normlong-range dependency
Prerequisites: RNN Architecture
Before we start

What you will learn

  • Derive why gradient magnitude scales with the recurrence's eigenvalues
  • Distinguish vanishing from exploding symptoms
  • Apply the standard mitigations in priority order
Introduction

What is this?

Backprop through time multiplies the gradient by the recurrence matrix's Jacobian every step, so its magnitude grows with the Jacobian's spectral radius (largest singular value) to the power T. Less than 1 → exponential vanishing (early context unreachable); greater than 1 → exponential blowing up (NaN). This single fact makes vanilla RNNs struggle with long context and motivates gated cells, normalization, clipping and skip connections.

Motivation

Why do you need it?

  • Long-range dependencies are the entire point of sequence models — vanishing kills them.
  • It explains the 'first 50 tokens are invisible' failure pattern you actually meet.
  • The same math underlies why depth hurts deep CNNs and drives residual/skip designs.
  • Every mitigation is a practical lever: clip numbers, gates keep radii near 1, skips give identity paths.
Think about it

Mental model

Imagine

Whispering through 50 people → shouting through open megaphone

Think of

Vanishing vs exploding recurrence

Whispers halve at each relay → a 50-relay message is lost (vanishing). A megaphone doubles each relay → the 50th blast deafens (exploding). Vanilla RNN recurrence is either bat of whispering or a megaphone, depending on the eigenvalue of Wh — and neither remembers anything useful.

Mechanics

How it works

  • The gradient for an early step contains ∏ over later Jacobians ∂h_t/∂h_(t-1).
  • If every Jacobian has norm < 1, the product → 0 exponentially as T grows.
  • If any eigenvalue exceeds 1, the product → ∞, exploding to NaN without clipping.
  • LSTM/GRU gating keeps dependence paths whose Jacobian norms stay ≈ 1 (the 'carry' cell state).
  • Clipping bounds updates; skip/residual paths let gradients bypass the recurrence chain.

Gradient magnitude vs distance back through time.

t = T (now)
Small Jacobians
Vanished
Large Jacobians
Gates/skips
Real world

In production

  • Practical speech/audio RNN towers use LSTMs/GRU instead of vanilla — the vanished-gradient fix is the whole reason.
  • Deep CNN spines add residual connections (skip paths) for the same class of problem.
  • Gradient clipping is on by default in most training libs for safety.
Guidelines

Best practices

  • Prefer LSTM/GRU over vanilla RNN unless demonstrations need the simplest cell.
  • Always clip gradients (clamp norm, e.g. 1.0) as a cheap safety net.
  • Truncate BPTT to e.g. 20-50 steps with carried state for long sequences.
  • Init recurrence shrink (e.g. Wh*0.1) if exploding persists before LR blame.
Watch out

Common mistakes

  • Blaming LR for explosive loss when the true cause is the recurrence matrix — clip and check norms first.
  • Using truncated BPTT without carrying state across chunks — you just cut context abruptly.
  • Clipping so aggressively that legitimately large early gradients are clipped into silence.
  • Forgetting the derivative is product-of-Jacobians, so 'the net is small' offers no protection.
Root cause analysis

Common issues

Error
Only the last ~10 tokens influence predictions
Root cause

Vanishing gradients over the unrolled depth

Fix

LSTM/GRU cell, skip connections, truncated BPTT with carry, or attention/transformer.

Error
NaN after a few hundred steps
Root cause

Exploding gradients (spectral radius > 1) — no clipping anywhere

Fix

Gradient clipping, shrink Wh init, add norm layers on states.

Error
Vanilla RNN can't learn a 2-year dependency
Root cause

Dependency length exceeds what chain multiplication preserves

Fix

Structural fix — gates/cell state explicitly designed to carry signal over long spans.

When things break

Debugging tips

  • Track per-layer gradient norms: flat-lower for early steps = vanilla vanish signature.
  • Print h_t norms; if they saturate at tanh bound at step 5 of 100, recurrence is unstable.
  • A truncated-BPTT gradient check (T=3 vs analytic) catches formula bugs fast.
  • When in doubt, swap the cell for an LSTM and see if the 'impossible' sequence becomes learnable.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The magnitude of a gradient through T recurrence steps scales like:

2

Gradient clipping fixes vanishing gradients.

Go deeper

Official documentation

On the difficulty of training RNNs
docs