Transformers
Intermediate 13 min read Transformers

Encoder-Decoder vs Decoder-Only

Two ways to organize a transformer

Two archetypes rule: encoder-decoders read a whole input then generate (translation, summarization), decoder-only models do everything by causal generation. Decoder-only won the platform war; knowing why reframes your choices.

encoder-decoderdecoder-onlyBARTBERTcausalbidirectional
Prerequisites: Positional Encoding
Before we start

What you will learn

  • Contrast encoder-decoder vs decoder-only
  • Match architecture to task type
  • Explain why decoder-only dominated for chat
Introduction

What is this?

Encoder-decoder transformers (e.g. BART, T5) encode a full input bidirectionally, then decoders generate causally — elegant for map-input-to-output jobs. Decoder-only models (GPT-style) generate everything causally, reading only what came before. Building outputs big, prompt-conditioned, and tool-rich made decoder-only the natural chassis for chat, agents and APIs.

Motivation

Why do you need it?

  • Encoder-decoders shine where the input is a fixed source to transform.
  • Decoder-only scales simply: one stack, one objective, long prompts as 'input'.
  • Chat = long prompt + generation — a decoder-only shape with no encoder needed.
  • Servers and fine-tunes are simplest on one unified decoder path.
Think about it

Mental model

Imagine

A translator desk vs a free writer

Think of

Encoder-decoder vs decoder-only

The translator studies the whole source twice (bidirectional) before dictating the translation. The writer composes forward, sentence by sentence, from the brief on their desk (the prompt). For interviews, email, drafts — the writer setup dominates.

Mechanics

How it works

  • Encoder stack: bidirectional self-attention over the whole input.
  • Cross-attention lets the decoder pull from encoder output per generated token.
  • Decoder-only: causal attention only; the prompt plays encoder's role.
  • Both end with softmax over vocab for next-token prediction.
  • Training objectives mirror these shapes (masked/full input vs next-token).

Two transformer shapes.

Encoder-DD
Source
Cross-attn
Decoder
Decoder-only
Real world

In production

  • Summarization/translation pipelines still favor encoder-decoder checkpoints for exact-source tasks.
  • Chat assistants, agents and tool-heavy apps universally chose decoder-only.
  • Hybrid uses: a decoder model plus a search/caller — not a second stack.
Guidelines

Best practices

  • Choose encoder-decoder when the task is source-to-text with a clear input object.
  • Choose decoder-only when the 'input' is a prompt mixture (state, tools, history).
  • Don't overinvest in 'which stack?' — the platform layers (RAG/tools) now cover most differences.
Watch out

Common mistakes

  • Assuming all 'transformers' are the same shape under the hood.
  • Picking an encoder-decoder checkpoint for chat and paying twice.
  • Ignoring that decoder-only is the ecosystem default for tool/agent support.
Root cause analysis

Common issues

Error
Translation with injected prompt corruption
Root cause

Cross-attention conflates source and prompt tokens

Fix

Keep the source block isolated and structured in the input.

Error
Chat tasks degrade on encoder-decoder
Root cause

Prompt-role architecture mismatch

Fix

Prefer decoder-only for conversational/prompt-conditioned generation.

When things break

Debugging tips

  • Encoder-decoder dropping source details → inspect cross-attention spread over source.
  • Decoder-only ignoring prompt mid-way → causal position effects (lost in the middle).
  • Both shapes 'work' for chat — benchmark YOUR task, not vibes.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Decoder-only models generate conditionally on…

2

Encoder-decoders read the source bidirectionally before generating.

Go deeper

Official documentation

Hugging Face transformer architectures
docs