Generative AI
Intermediate 16 min read Generative AI

Language Model Training

The pretrain → align pipeline that built the LLM era

LLMs are trained to maximize the probability of the next token over trillions of tokens, then aligned (instruction tuning, RLHF) to follow instructions. Compute and data — not architecture flair — dominate the recipe.

pretrainingnext-tokencross-entropydata scalinginstruction tuningRLHFalignment
Prerequisites: Generative Models Overview Decoder-Only (GPT)
Before we start

What you will learn

  • Explain next-token maximum likelihood training
  • Order the pipeline: pretrain → SFT → RLHF
  • Relate data quantity and compute to model quality (scaling laws)
Introduction

What is this?

An LM is trained by maximizing the log-likelihood of the next token given the prefix across the whole corpus — a giant softmax over the vocabulary at every position. Pretraining uses web-scale text and enormous compute. Then alignment stages re-pretrain the same weights toward instruction-following: supervised fine-tuning (SFT) on human/curated demos and preference optimization (RLHF/DPO) that nudges outputs toward preferred answers. 'Scaling laws' say loss reliably drops with more data, parameters and compute.

Motivation

Why do you need it?

  • Next-token prediction over trillions of tokens is the simplest objective that yields useful competence.
  • Unlabeled web text is nearly free; labels for instruction-following are expensive — separate the stages.
  • Alignment converts 'predict the next token' into 'do what the user asks'.
  • Scaling laws make resource planning quantitative: more data/compute/props predictably improve quality.
Think about it

Mental model

Imagine

An apprentice studying every book ever written, then taking etiquette lessons

Think of

Pretraining then alignment

Years of reading give mastery (the distribution over language). Etiquette lessons (SFT/RLHF) then teach how to actually behave when addressed — useful, safe, on-task answers — without re-reading the library.

Mechanics

How it works

  • Tokenize the corpus; each training example is a sequence with input tokens shifted by one as targets.
  • Loss = average cross-entropy between the predicted distribution and the true next token.
  • Optimization over parameter count and tokens: scaling laws fit loss ≈ a·(compute)^-b curves.
  • SFT: fine-tune on instruction→response pairs with standard cross-entropy, low LR, few epochs.
  • Preference tuning: train a reward model on chosen/rejected pairs, then optimize the LM toward it (RLHF), or use direct preference optimization (DPO) in one loss.

Two-stage training: likelihood, then alignment.

Web corpus
Next-token CE
Base model
SFT demos
RLHF / DPO
Real world

In production

  • Base models power token-completion features; chat models power assistants.
  • SFT-only small models run on-prem for cost and compliance.
  • Continual/task fine-tuning adapts a model to enterprise domains (law, code, medicine).
Guidelines

Best practices

  • Buy pretraining compute; spend your own data/compute budget on SFT (the highest ROI for a product).
  • Keep SFT datasets small and high-quality — thousands of curated examples beat millions of scrappy ones.
  • Track loss, eval perplexity and task checkpoints; use a small eval set consistently across runs.
  • Prefer measured downstream gains over 'more training steps' — diminishing returns hit fast.
Watch out

Common mistakes

  • Retraining from scratch when a strong open model plus SFT would cost 100x less.
  • Using the same LR as pretraining during SFT and destroying learned knowledge.
  • Treating RLHF as a fix-all: a broken base model stays broken.
  • Measuring alignment by loss only, ignoring whether outputs match user intent.
Root cause analysis

Common issues

Error
SFT loss plateaus high
Root cause

LR/deep LR decay too aggressive or dataset mislabeled

Fix

Lower LR, increase warm-up, and audit the instruction dataset for contradictions.

Error
RLHF rewards rise but outputs get repetitive
Root cause

Reward hacking — model exploits the reward model

Fix

Strengthen KL penalty, add per-turn reward guards, and sample-diversity metrics.

Error
Fine-tuned model forgets general ability
Root cause

Catastrophic forgetting from too many steps / too few diverse data

Fix

Add a small general-knowledge set to SFT and early-stop on a general eval.

When things break

Debugging tips

  • Overfit one training example first (loss → near 0) to prove the pipeline, then scale data.
  • Compare SFT vs base-model outputs on the same prompts to quantify alignment gain.
  • Watch reward-model score and KL penalty together during RLHF — drifting KL signals instability.
  • Log tokenizer outputs early; subtle tokenization bugs silently waste epochs.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The core pretraining objective of a decoder-only LLM is:

2

Instruction-tuning stage usually uses far fewer tokens than pretraining.

Go deeper

Official documentation

Chinchilla scaling laws
docs