RNN
Beginner 13 min read RNN

Sequence Modeling

When the order of inputs is the signal

Sequence modeling predicts from ordered inputs — time series, text, audio, events. Order is information, so the model must respect it: what happened before shapes what happens now.

sequential datatime seriesRNNsequencecontext windoworder
Prerequisites: Deep Learning Basics
Before we start

What you will learn

  • Recognize which problems are sequential and why order matters
  • Distinguish fixed-window models from stateful recurrent models
  • Frame a sequence task as predict-next vs classify-sequence
Introduction

What is this?

Sequence modeling handles data where inputs arrive in an order and that order carries meaning — a sentence, a heartbeat trace, a purchase history. Predictions depend on positions relative to each other, so a good sequence model consumes the whole ordered input (or a window of it) instead of treating records as independent. Classic approaches slide a fixed window into an MLP; recurrent models carry state across the entire sequence.

Motivation

Why do you need it?

  • Order matters: 'schedule the meeting' vs 'the meeting schedule' — shuffling destroys the problem.
  • Context grows: one word's meaning depends on the sentence, the sentence on the conversation.
  • Forecasting is inherently sequential — next value given all previous values.
  • Choosing window-vs-recurrent is a real architecture decision with capacity and cost implications.
Think about it

Mental model

Imagine

Reading a book forward only, then deciding the next word

Think of

Sequential prediction with a recurrent model

You keep a running understanding (state) built from every page so far; you don't re-read the whole book each page. That state — not just the last page — informs 'what happens next'.

Mechanics

How it works

  • Represent the sequence as tokens or numeric features in order: x1, x2, ..., xT.
  • A fixed-window model takes the last k values as one vector — simple, bounded context.
  • A recurrent model processes one token at a time, updating a hidden state h_t from h_(t-1) and x_t.
  • The state compresses all past context ('what I've seen so far') into a fixed-size vector.
  • Outputs may be per-step (predict next token) or after the whole sequence (classify the sequence).

Fixed window vs recurrent state.

Inputs
Window model
Recurrent state
Predict
Output
Real world

In production

  • Time-series forecasting: stock sims, demand, energy — predict the next step from history.
  • Speech/audio tagging: classify segments or phonemes from ordered frames.
  • Text as sequence: sentiment, translation, language modeling — every token depends on its context.
Guidelines

Best practices

  • Encode order explicitly: positional features, lags, or careful tokenization.
  • Choose the model on the needed context length: window models for short, bounded context.
  • Normalize per-sequence or per-batch rather than globally when scale varies.
  • Report metrics at the granularity of your decision (per-step accuracy vs whole-sequence)
Watch out

Common mistakes

  • Shuffling training batches before deciding order is irrelevant — destroys sequential structure.
  • Sliding windows too small to capture the actual dependencies.
  • Ignoring event-time vs processing-time when the sequence comes from real-world logs.
  • Assuming 'sequence model' must be recurrent — modern transformers often win on long context.
Root cause analysis

Common issues

Error
Model ignores later context — long pauses all look the same
Root cause

Window/state too short to carry the relevant history

Fix

Widen window, add lag features, or switch to a recurrent/attention model.

Error
Accuracy great on train, poor on online data
Root cause

Online stream arrives with different cadence/order — event time unused

Fix

Use event time + sequence ids and validate on a real-time replay.

Error
Every prediction is the global mean/latest value
Root cause

Imbalanced sequence lengths or batch normalization across sequences

Fix

Mask padded regions and use per-sequence normalization.

When things break

Debugging tips

  • Plot the sequence and label distribution; most 'model errors' are mislabeled/truncated sequences.
  • Check context length: if accuracy jumps when you widen a window, the dependency was longer than you thought.
  • Compare windowed-MLP vs recurrent baselines before blaming the architecture.
  • Verify ordering survived preprocessing (sort/merge shuffles are a classic silent bug).
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

What is the key signal a sequence model must respect?

2

A fixed sliding-window MLP can never see context before its window.

Go deeper

Official documentation

The Unreasonable Effectiveness of RNNs
docs