Transformers
Intermediate 15 min read Transformers

Encoder-Only Transformers (BERT)

Bidirectional context, pretrained by masking

BERT stacks bidirectional transformer encoders and pretrains with masked-language modeling (predict masked tokens). Its dense, context-aware embeddings are the foundation for classification, retrieval and fine-tuning.

BERTencoder-onlymasked language modelingMLMpretrainingembeddingsfine-tune
Prerequisites: Transformer Architecture
Before we start

What you will learn

  • Describe MLM pretraining and the [MASK]/next-sentence objects
  • Explain why bidirectional context makes better embeddings
  • Know how BERT-style models are fine-tuned for tasks
Introduction

What is this?

BERT is an encoder-only transformer pretrained on huge text with masked language modeling: random tokens are replaced by [MASK] and the model must predict the original token from bidirectional context. This learns deep, context-dependent token representations. A classifier head is added on top of the [CLS] token (or pooled tokens) and fine-tuned on a small labeled dataset for downstream tasks.

Motivation

Why do you need it?

  • Bidirectional context is what embeddings need most — 'bank' resolves from both its left and right.
  • Pretraining on unlabeled text transfers; fine-tuning needs only hundreds-to-thousands of labels.
  • Dense embeddings serve retrieval, clustering and classification beyond text classification.
  • It set the encoder-side pattern that most non-generative NLP stacks still follow.
Think about it

Mental model

Imagine

A vocab-wide apprentice trained by dictation with missing words

Think of

BERT's masked language modeling

The apprentice is repeatedly given sentences with a word blanked out and must infer the missing word from the surrounding sentence — both what came before AND after. Practicing millions of such dictations builds a fine internal dictionary, which then slots into any office specialty (the fine-tuned tasks).

Mechanics

How it works

  • Tokenize input (often WordPiece/subword), add [CLS] + [SEP], sum token+segment+position embeddings.
  • Mask ~15% of tokens: 80% → [MASK], 10% → random token, 10% → unchanged (robustness).
  • Objective: predict the masked token ids from hidden states — a large vocabulary softmax.
  • Stacked bidirectional self-attention gives each token full left+right context.
  • Fine-tune: freeze/keep pretrained backbone, add a head, train with small LR on the task dataset.

Masked input through encoder blocks to a token prediction head.

The [MASK] is [MASK]
Stacked encoders
Hidden states
LM head
CLS head
Real world

In production

  • Named-entity recognition, sentiment/offense classification — add a head, fine-tune.
  • Sentence embedding & retrieval: mean/CLS pooling feeds vector databases for RAG/search.
  • Document classifiers where a single label per snippet is required at scale.
Guidelines

Best practices

  • Fine-tune at a small LR (e.g. 2e-5 to 5e-5) and a few epochs — BERT heads adapt fast.
  • Prefer mean-pooling of the last hidden layer over raw CLS for retrieval embeddings.
  • Match the tokenizer/model exactly; never train a custom tokenization on BERT 'as-is'.
  • Use gradient accumulation when labeled data is tiny; watch for catastrophic forgetting.
Watch out

Common mistakes

  • Using BERT for generation — it has no autoregressive decoder.
  • Forgetting that [MASK] is pretraining-only; never mask at inference.
  • Fine-tuning with a huge LR and destroying the pretrained features.
  • Pooling all tokens including padding when building sentence vectors — normalize like the model expects.
Root cause analysis

Common issues

Error
Fine-tuning destroys pretrained quality
Root cause

LR too large on backbone or unbacked head

Fix

Lower LR, warm up, use freezes on lower layers or larger batches/accumulation.

Error
Retrieval embeddings all collapse to one vector
Root cause

Mean-pooling over padding or head training on too-similar pairs

Fix

Attention/CLS pooling, padding masking in pooling, and harder negatives.

Error
Long docs lose coherence for classification
Root cause

Context window truncates the tail

Fix

Chunk + aggregate logits (hierarchical) or switch to encoder-decoder sum/mean heads.

When things break

Debugging tips

  • Sanity: mask known fill-in-the-blank: 'Paris is the capital of [MASK]' should predict France.
  • Check CLS-vector norms: collapsed/degenerate norms often mean pooling/batch issues.
  • Compare fine-tune vs frozen embeddings on a small task — that attributes the gain to fine-tuning.
  • Log tokenizer special-token ids to confirm no accidental [MASK] leakage into inference.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

BERT's primary pretraining objective is:

2

You should keep [MASK] tokens in inference inputs to improve accuracy.

Go deeper

Official documentation

BERT paper
docs
Hugging Face transformers doc
docs