Pre-training & the Base Model
Reading the internet, once, at planetary scale
Pre-training learns language from next-token prediction over enormous corpora, producing a 'base' checkpoint — general, unaligned, and dangerous-ish before post-training. Knowing this split explains what you inherit and what you must add.
What you will learn
- Describe pre-training data and objective
- Differentiate base vs aligned checkpoints
- Choose base vs instruction checkpoint for a task
What is this?
Pre-training is the gigantic next-token-prediction phase over curated web/text corpora (billions of tokens), yielding a base checkpoint — breathtaking at continuation, unsocialized as a product. Post-training (instruction tuning, RLHF-style alignment, safety) turns that base into an assistant. Platforms ship aligned models; you mostly inherit the whole stack via APIs.
Why do you need it?
- The base model defines raw capability; alignment defines usability and safety.
- Base checkpoints are the accurate raw 'continuator'; aligned ones obey prompts.
- Fine-tuning for your task starts from a checkpoint that already 'knows' the domain syntax.
- Understanding the split demystifies 'why did the model do that?' — often it's the base, not injection.
Mental model
Learning a language fluently vs becoming a useful guide
Pre-training + alignment
Pre-training teaches the language (structures, facts, style). Post-training teaches how to guide a tourist (follow the itinerary, don't rant at strangers, say 'I don't know'). You inherit both when you call an aligned API, but you're buying the guide, not the linguist.
How it works
- Corpus: curated text sweep (web, books, code) filtered and deduplicated.
- Objective: predict the next token; learn syntax, facts, reasoning priors.
- Base checkpoint: next-token machine, no prompt discipline.
- Post-training: instruction tuning + preference/alignment + safety filters.
- You consume the aligned result via APIs or download base/adapted open checkpoints.
The two-phase life of a model.
In production
- Choosing checkpoints: aligned API models for product defaults, base open models for supervised fine-tunes you align yourself.
- Domain APIs (legal/medical) profit from fine-tuning an aligned checkpoint with domain data.
- Eval story: your benchmarks measure inherited pretrain knowledge + your post-training differences.
Best practices
- Prefer aligned model checkpoints for anything user-facing.
- Audit base-model outputs vs aligned on your risky tasks — differences are your safety surface.
- Version checkpoints: the base identity governs capability cliffs, not day-2 prompts.
Common mistakes
- Using a raw base model as a product (hallucination and incoherence galore).
- Assuming fine-tuning 'resets' pretrain pathologies — it doesn't.
- Measuring 'model upgrade' quality on 10 prompts.
Common issues
Model says clearly-wrong things fluentlyBase capability vs grounding gap
Ground via retrieval/validators, steer via better prompts, or use a stronger aligned checkpoint.
Fine-tuned model lost general abilityCatastrophic forgetting during tuning
Blend general examples, regularize, and use LoRA with measured eval.
Debugging tips
- Clients get textbook-but-wrong prose → base-quality artifacts; add grounding/validators.
- Model parrots harmful stereotypes → alignment gap; avoid or hard-filter the domain.
- Code tasks worse after a 'fine-tune' → you broke base capability with bad data.
Interview checkpoint
Quick quiz
What is pre-training's objective?
A base checkpoint is safe to ship as a user-facing assistant out of the box.