RNN
Advanced 15 min read RNN

Sequence-to-Sequence Patterns

Map one sequence to another

Seq2seq learns an input sequence → output sequence mapping: encoder reads one sequence into state, decoder generates the other. That framework powers translation, summarization and more — and attention fixed its context bottleneck.

seq2seqencoder-decoderencoder statedecoderteacher forcingattentioncontext vector
Prerequisites: RNN Architecture
Before we start

What you will learn

  • Describe the encoder-decoder split and the context vector
  • Explain teacher forcing and its training-time role
  • Explain why attention replaced the single context vector
Introduction

What is this?

Sequence-to-sequence models map between ordered inputs and ordered outputs — French sentence → English, audio frames → words, code → summary. An encoder reads the input into a hidden/context vector (through a recurrent or attention stack), and a decoder generates the output one token at a time, conditioned on that context and its own previous outputs. During training the decoder often takes the true previous token (teacher forcing) to learn faster.

Motivation

Why do you need it?

  • It generalizes connectionist sequence problems beyond one-output labeling.
  • Separating encoder/decoder means each side can model its own semantics.
  • Attention over the encoder states was the breakthrough that let decoders 'look back', fixing long-context loss.
  • It's the same skeleton modern encoder-decoder transformers (BERT-encoders + GPT-decoders) still use.
Think about it

Mental model

Imagine

An interpreter and a writer

Think of

Seq2seq encoder + decoder

The interpreter (encoder) listens to the full input and distills the gist (context vector); the writer (decoder) turns that gist into polished output, checking back at the interpreter (attention) when a detail matters.

Mechanics

How it works

  • Encoder processes the input, producing hidden states; the last state (or attention over all states) becomes context.
  • Decoder is seeded with the context vector and a start token.
  • At each step it outputs a token distribution over the vocabulary; the chosen token feeds the next step.
  • Teacher forcing: during training feed the TRUE previous token instead of the model's sample, for faster, stable learning.
  • Attention lets the decoder weight salient encoder states rather than compressing everything into one vector.

Encoder → context → decoder rollout.

Input tokens
Encoder
Context
Decoder
Output tokens
Real world

In production

  • MT systems (pre-transformer) were seq2seq RNNs with attention.
  • Speech-to-text decoders still stack RNN/tranformer decoders over acoustic encoders.
  • Summarization and dialogue slots reuse encoder-decoder structure with attention decoders.
Guidelines

Best practices

  • Use attention over encoder states; never rely on a single last-state vector for long inputs.
  • Bucket by output length too — decoder cost grows with generation, not just input.
  • During eval, decode greedily or beam-search; teacher forcing distorts eval likelihood.
  • Schedule sampling (anneal teacher forcing) improves robustness to its own errors.
Watch out

Common mistakes

  • Teacher-forcing forever: the model never learns to recover from its own mistakes at inference.
  • Using the last-state-only context on a 200-token input — vanishing context.
  • Feeding the wrong start token (e.g. BOS vs EOS) and off-by-one crashes at eval.
  • Decoding with a batch of padded sequences and sampling pad tokens as valid output.
Root cause analysis

Common issues

Error
Decoder repeats one token forever
Root cause

Degenerate generation loop worsened by no attention/masking or softmax temperature 0

Fix

Add length penalty/masks, sampling with temperature, or stop-token detection.

Error
Very long inputs degrade translation
Root cause

Single context vector can't hold 500 tokens' meaning

Fix

Attention over encoder states, or a transformer encoder.

Error
Train loss low, generation garbage
Root cause

Teacher forcing hides decoding mistakes; model never self-corrects

Fix

Apply scheduled sampling and evaluate with real rollout decoding.

When things break

Debugging tips

  • Print decoder step logits for the first 5 tokens — wrong BOS or vocab ids surface immediately.
  • Check that attention weights over encoder states sum row-wise to 1 (softmax bug).
  • Overfit a tiny 3-pair corpus: generation should copy/paraphrase to prove plumbing.
  • Compare greedy vs beam on a fixed example before tuning beam width blindly.
Interviews

Interview checkpoint

Practice

Hands-on lab

Build a Tiny Seq2Seq in Numpy
30 min · advanced
In progress
Objectives
  • Implement encoder and decoder RNNs by hand
  • Wire a context vector and a teacher-forced decoder
  • Verbalize the seq-to-seq protocol on a toy copy task

Build a Tiny Seq2Seq in Numpy

30 min · advanced

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

In seq2seq, the decoder at each step conditions on:

2

Teacher forcing continues to feed true tokens during inference.

Go deeper

Official documentation

Neural Machine Translation by Jointly Learning to Align (attention)
docs