LSTM
Intermediate 14 min read LSTM

Bidirectional & Stacked RNNs

Seeing both sides of each position

Bidirectional RNNs read a sequence both forward and backward and concat the hidden outputs, so each position sees full context. Stacking rows of RNN cells adds hierarchical abstraction.

bidirectionalbackward passconcatstacked RNNcontextencoder
Prerequisites: LSTM Intuition Sequence-to-Sequence Patterns
Before we start

What you will learn

  • Explain how bidirectional layers give each step full-sequence context
  • Describe stacked layers and their feature-hierarchy effect
  • Know when bidirectionality is invalid (online/stream decoding)
Introduction

What is this?

A bidirectional RNN runs two independent RNNs — one forward (x1→xT), one backward (xT→x1) — and concatenates their hidden states at each position, so step t sees left AND right context. Stacked RNNs put several RNN layers in sequence, each feeding the next, gaining hierarchical abstraction like deeper CNNs. Bidirectional+stacked is the classic encoder for offline tagging/classification.

Motivation

Why do you need it?

  • Many tasks decide a token from BOTH sides: 'bank' needs the earlier and later words — and 'Mr. Bank' differs from 'hard-bank loan'.
  • Concat-doubles the representation at no extra data cost; big encoder gains for classif/transl.
  • Stacked depth composes level-of-abstraction the same way CNNs do.
  • Online/streaming can't use backward — knowing this avoids a whole bugs class.
Think about it

Mental model

Imagine

Two proofreaders covering each line from opposite ends

Think of

Bidirectional context

Reader A scans left→right, Reader B right→left; at every meeting point their combined notes (concatenated h) say more about the line than either alone. But you can't run this proofret inside a live whisper session that must speak immediately.

Mechanics

How it works

  • Forward layer: h_f(t) from x1..xt; backward layer: h_b(t) from xT..xt.
  • Concatenate: h_t = [h_f(t); h_b(t)] — double width feeds the next layer/head.
  • Stacked: layer 1 outputs feed layer 2's cells; each layer adds a level of sequence abstraction.
  • Final readouts (classify/tag) consume the top layer's concat states.
  • Params double vs single-direction; memory per step doubles; latency 2x a simple pass.

Bidirectional layer: forward + backward concat.

x1 → ...
← xT ...
Concat
Head
Full context
Real world

In production

  • NLP tagging (NER, POS, chunking) and sentence classifiers use bidirectional encoders heavily.
  • Offline ASR+MT encoders read the full utterance bidirectionally for better representation.
  • Stacked+bidi recurrent encoders appear in older but production-widespread medical/signal pipelines.
Guidelines

Best practices

  • Use bidirectionality ONLY for offline (full-sequence available) tasks.
  • Start with one bidi layer on top of a unidirectional stack — often the sweet spot.
  • Remember concat doubles the top-layer hidden size for the head's input dim.
  • For streaming, keep unidirectional (or causal attention) — never backward-peek at inference.
Watch out

Common mistakes

  • Using bidi in live streaming and silently 'rewinding' the future — a correctness bug.
  • Forgetting the doubled width when sizing the head/softmax input.
  • Stacking 4 bidi layers 'for depth' and blowing memory for marginal gains.
  • Backward layer using the same weight matrix as forward (not two copies) in hand-rolled code.
Root cause analysis

Common issues

Error
Backward direction trains as zeros
Root cause

Hidden init or input reversed; or optimizer sees NaN from reversed padding

Fix

Reverse correctly with packed sequence handling for variable lengths.

Error
Online serving breaks after adding bidi
Root cause

Model requires future inputs that a stream hasn't produced

Fix

Keep the online model unidirectional; bidi only for offline scoring.

Error
Memory doubles and training slows
Root cause

Two full direction states + concat

Fix

Use a single bidi layer + unidirectional stack, or switch to attention for long-but-offline.

When things break

Debugging tips

  • Sanity: a bidi model should score higher than unidirectional on offline NER/data with meaningful context.
  • Check that the backward layer gets x_T first (reverse order), not x_1.
  • Print h_f and h_b norms separately — an exploded direction pinpoints that half of the cell.
  • Latency test: bidi adds ~2x time per pass; solve the loss first, then cocktail with depth.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

In a bidirectional RNN, the hidden state at position t is:

2

Bidirectional RNNs are safe for live streaming inference.

Go deeper

Official documentation

PyTorch bidirectional flag docs
docs