RNN
Advanced 15 min read RNN

Training RNNs

Backprop through time, done carefully

RNN training is BPTT: unroll, backprop through T steps, gradient-clip, repeat. Practical craft lives in truncated BPTT, length bucketing, state carry, and learning-rate hygiene on long sequences.

BPTTtruncated BPTTgradient clippingstate carrysequence lengthpackinglearning rate
Prerequisites: RNN Architecture Vanishing Gradients
Before we start

What you will learn

  • Describe full vs truncated BPTT and state carrying
  • Apply gradient clipping and bucketing correctly
  • Diagnose long-sequence training instabilities
Introduction

What is this?

Training an RNN means unrolling its graph over the sequence and backpropagating through every step — BPTT — then updating the shared weights. Truncated BPTT unrolls only a window (e.g. 20 steps) while carrying the state across the boundary to keep long-range memory without a monster graph. Practical training adds gradient clipping, length bucketing/packing, and LR schedules tuned for deep effective depth.

Motivation

Why do you need it?

  • Full BPTT over 1,000 tokens is memory-prohibitive and numerically awful — truncation is standard.
  • The effective model is as deep as your sequence length, so LR/grad hygiene matters more than in feedforward nets.
  • Getting training right is usually 80% of RNN success — cells are secondary.
  • Modern practice (attention) borrows the same discipline with different memory patterns.
Think about it

Mental model

Imagine

Rowing a boat while noting every rope pull ever

Think of

Rethinking BPTT truncation

Rowing a long trip, you only need the last strokes' feedback to steer, but you must remember upstream current (state carry). Full-BPTT would re-evaluate every pull since the marina; smart rowing carries the boat's state forward while adjusting only from recent strokes.

Mechanics

How it works

  • Full BPTT: unroll all T steps, compute loss, backprop through all of them, update once.
  • Truncated: unroll W steps, don't backprop farther, but KEEP h carried from the previous W-window.
  • Packing sorts by length and groups, so batches don't waste padding compute.
  • Gradient clip by global norm (e.g. 1.0) to tame the exponential growth.
  • LR: start small, use schedule/adaptive optimizer, and re-checkable with range tests per cell type.

Truncated BPTT with state carry across windows.

Window 1
Truncate
Window 2
BPTT
Update & repeat
Real world

In production

  • Speech recognition towers train with truncated BPTT + state carry on Windows of acoustic frames.
  • Time-series RNN products pack/bucket by forecast horizon to keep batch compute honest.
  • RL-from-history agents train recurrent critics with mixed BPTT using carried states.
Guidelines

Best practices

  • Carry the state across truncation boundaries; silent state-reset loses long memory.
  • Bucket by length and mask padded steps in the loss.
  • Debug with LSTM/GRU cells first, then try vanilla only for short sequences.
  • Use gradient cropping and watch per-layer grad norms after each epoch.
Watch out

Common mistakes

  • No state carry — equivalent to a fresh RNN every window.
  • Applying batch norm to a batched unroll where padding injects fake steps.
  • Forgetting mask → gradients flow into padding positions.
  • Small LR then blame the model: LR-range-test with the actual cell length first.
Root cause analysis

Common issues

Error
Long sequences barely learn but short ones do
Root cause

Truncation window too small or state not carried

Fix

Lengthen window, ensure carry, and switch cell to LSTM/GRU.

Error
Loss jumps every ~window length
Root cause

State gets reset at truncation boundaries (forgot carry)

Fix

Pass the computed h from the previous window into the next batch's initial state.

Error
NaN right after gradient clipping is enabled
Root cause

Clip applied to wrong object (per-parameter instead of global norm) or overflow before clip

Fix

Clip the full gradient vector norm; move clip before optimizer; scale to fp32 sooner.

When things break

Debugging tips

  • Print grad-norm by time-window: if window 1 ≈ 0 and window 4 healthy, vanishing in action.
  • Overfit a 3-sequence set to validate truncation/carry plumbing before larger runs.
  • Compare train with truncation=100 vs 20; behavior change shows where the cut bites.
  • Confirm packed/padded loss explicitly excludes pad ids via a mask matrix.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Truncated BPTT refers to:

2

Padded sequence positions should contribute to the loss.

Go deeper

Official documentation

PyTorch packing utilities
docs