LSTM
Intermediate 15 min read LSTM

LSTM Intuition

A learnable memory with a gradient highway

LSTMs give a recurrent network an explicit memory cell with gates that decide what to keep, what to write, and what to output. That cell-state highway fixes vanilla RNNs' vanishing gradients.

LSTMcell stateforget gateinput gateoutput gatememorygating
Prerequisites: RNN Architecture
Before we start

What you will learn

  • Describe the cell state and the three gates at a conceptual level
  • Explain why gates keep gradients from vanishing
  • Read an LSTM diagram and identify each gate's input
Introduction

What is this?

An LSTM extends an RNN with a second line of state: the cell state c_t, a conveyor belt that runs straight through the network, modified additively. Three gates — forget, input, output — are learned sigmoid switches deciding what the cell remembers, what new info to write in, and what to expose to the hidden state. Because the cell updates by addition (not multiplication), gradients flow backward almost undiminished over hundreds of steps.

Motivation

Why do you need it?

  • Vanilla RNNs multiply state each step — forgetting everything after ~20 steps.
  • The cell-state path's Jacobian is ≈ 1 when the forget gate is open, solving vanishing gradients structurally.
  • Gates are differentiable: the whole thing still trains end-to-end with backprop.
  • LSTMs made real sequence products (speech, forecasting, early NLP) actually learnable.
Think about it

Mental model

Imagine

A warehouse with three checked doors

Think of

An LSTM cell

The conveyor belt (cell state c) runs through the warehouse. Door 1 (forget) decides what to discard; door 2 (input) brings in new stock; door 3 (output) shows the bay door what's worthy to ship. The belt itself never gets bottlenecked — stuff flows through regardless of what moves.

Mechanics

How it works

  • Forget gate: f_t = σ(W_f·[h_(t-1), x_t] + b_f) — weights how much of c_(t-1) to keep.
  • Input gate & candidate: i_t and ĉ_t — decide what NEW info to write into c_t.
  • Update: c_t = f_t ⊙ c_(t-1) + i_t ⊙ ĉ_t — the additive highway.
  • Output gate: o_t = σ(...) selects what to reveal; h_t = o_t ⊙ tanh(c_t).
  • Readouts use h_t; the cell stays internal memory.

An LSTM cell: belt + three gates.

c_(t-1)
Forget gate
Input gate
c_t
Output gate
Real world

In production

  • Anything a vanilla RNN would do, an LSTM does better in practice — speech, forecasting, activity recognition.
  • Deep stacks of LSTMs with attention were the pre-transformer state of the art in MT and ASR.
  • They remain the default stateful cell when transformers are too heavy for the context/latency budget.
Guidelines

Best practices

  • Init the forget-gate bias high (e.g. +1) so the cell starts 'keeping everything' and learns what to drop.
  • Use LSTM when you need long spans (minutes of audio, hours of telemetry) with limited state budget.
  • Pair with layer-norm on the cell state (LayerNormLSTM) for harder sequences.
  • Prefer GRU when compute/params are tight and your data is moderate — it's a lighter gate set.
Watch out

Common mistakes

  • Believing LSTM cures explosive gradients — it needs clipping like any recurrent net.
  • Zeroing the forget gate repeatedly by a badly-tuned init — model defaults to amnesia.
  • Using the cell state directly for classification instead of h_t (head should consume h).
  • Adding more width when a deeper LSTM (2-3 layers) is what the problem wants.
Root cause analysis

Common issues

Error
LSTM behaves like a 5-step RNN on a 400-step problem
Root cause

Forget-gate collapses to zero (amnesia) early — carrying nothing

Fix

Higher forget-bias init (+1), reduce LR, or clamp forget low-bounds.

Error
Cell state norms climb forever
Root cause

Input flow exceeds forget flow — writing faster than forgetting

Fix

Add weight decay/clipping on c, grow width or sequence-bucket longer context.

Error
First tokens still ignored
Root cause

Gradient-tracking too shallow: truncation or shared init erases early path

Fix

Carry state, lengthen truncation, lower LR for the deep chain.

When things break

Debugging tips

  • Print mean forget-gate value over training: if it collapses toward 0.1, the cell is 'forgetting too hard'.
  • Compare LSTM vs GRU vs vanilla on short and long sequences — fastest diagnosis of gate usefulness.
  • When gradients still vanish, check hidden init and add layer-norm on inputs/cell.
  • Truncate BPTT leniently (e.g. 50-100) since the cell path tolerates deeper chaining.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

What does the forget gate decide?

2

LSTMs are immune to gradient instability.

Go deeper

Official documentation

Long Short-Term Memory (Hochreiter & Schmidhuber)
docs