RNN
Intermediate 15 min read RNN

RNN Architecture

A network unrolled over time with shared weights

An RNN feeds each input together with the previous hidden state through a weight-shared cell, carrying information across steps. Unrolled over time it's a deep shared-weight network — with all the gradient implications.

RNNhidden statecellshared weightsunrollingtanhtime steps
Prerequisites: Sequence Modeling
Before we start

What you will learn

  • Describe the RNN cell update h_t = σ(Wx·x_t + Wh·h_(t-1) + b)
  • Explain weight sharing across time steps
  • Unroll a 3-step RNN and identify its depth
Introduction

What is this?

A recurrent network uses one cell applied identically at every time step. At time t the cell combines the current input x_t with the previous hidden state h_(t-1): h_t = tanh(Wx·x_t + Wh·h_(t-1) + b). The same Wx, Wh and b are reused for every step — that's the 'recurrent' weight sharing. Unrolled, an RNN over T steps behaves like a T-layer network whose layers all share weights.

Motivation

Why do you need it?

  • Weight sharing lets an RNN handle variable-length sequences with a fixed number of parameters.
  • The state h_t compresses the whole past into a fixed vector the model can read.
  • Unrolling the recurrence is how backprop through time (BPTT) computes gradients.
  • Because the same cell repeats, the number of steps T is your effective depth — and that's where gradients struggle.
Think about it

Mental model

Imagine

A runner handing off a baton at each lap marker

Think of

RNN state flowing through steps

The same runner (the shared cell) takes the baton (state) at each marker, reads where they are (input), and passes the baton forward — the baton is the accumulated knowledge, and the whole run depends on the same consistent stride (weights).

Mechanics

How it works

  • Initial state h_0 (often zeros) starts the chain.
  • At each t: h_t = tanh(Wx·x_t + Wh·h_(t-1) + b); h_t is the new state.
  • Outputs are read from h_t by an output layer (per-step) or from the final h_T (sequence label).
  • Unrolling: the computation graph is T identical cells deep — total depth T.
  • Training: backprop through time (BPTT) computes gradients by unrolling and chaining through all T steps.

An RNN unrolled for 3 steps with shared weights.

x1 → h1
x2 → h2
x3 → h3
Readout
Backprop
Real world

In production

  • Classic building blocks (before transformers) for speech-to-text stateful decoders.
  • Character/music generators with vanilla RNNs still appear in lightweight demos.
  • RL wrappers that compress observation history into a state vector.
Guidelines

Best practices

  • Truncate BPTT to a manageable window while keeping states carried across batches.
  • Use tanh activation (saturating, but with well-understood gradients) inside the cell.
  • Initialize Wh to stay contractive (small-scale) to tame early recurrence blow-ups.
  • Package sequences by length (bucket padding) to avoid wasting the shortest batch on the longest pad.
Watch out

Common mistakes

  • Sharing weights but training step-dependent copies (bug: not untying correctly in custom loops).
  • Feeding un-padded variable-length sequences into a fixed shaped matrix.
  • Not masking padded positions, so the loss 'learns' padding as context.
  • Ignoring T-depth: a 500-token sentence = a 500-layer net's gradient path.
Root cause analysis

Common issues

Error
State norm explodes over long sequences
Root cause

Recurrence matrix spectral radius > 1 amplifies signal each step

Fix

Smaller Wh init, gradient clipping, or batch/group-norm on h; prefer LSTM/GRU cells.

Error
Early steps contribute nothing (loss fixed by last steps only)
Root cause

Vanishing gradients through T unrolls

Fix

Truncated BPTT + carry, better init, or a cell with gates (LSTM/GRU).

Error
Padding rows dominate the batch stats
Root cause

No length bucketing or masking

Fix

Sort/bucket by length and mask pads in both loss and normalization.

When things break

Debugging tips

  • Print h_t norms per step: explosion over steps signals a Wh-scale/LR problem.
  • Run a 2-step toy and check the gradient via finite differences before scaling length.
  • Masked loss? Confirm padding tokens contribute zero to loss, or pad-error dominates.
  • Compare T=5 and T=50 behavior — the pain switching tells you gradient difficulty.
Interviews

Interview checkpoint

Practice

Hands-on lab

A From-Scratch RNN Stepwise
25 min · intermediate
In progress
Objectives
  • Implement the RNN cell update in numpy
  • Unroll 3 steps and confirm state norms behave
  • Compare with a framework RNN output on matching weights

A From-Scratch RNN Stepwise

25 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which parameters are shared across all time steps of an RNN?

2

An RNN's computational depth grows with sequence length.

Go deeper

Official documentation

PyTorch RNN docs
docs