What Language Models Are
Probabilistic text machines
Large language models predict the next token autoregressively; generation is sampling that prediction, and 'reasoning' emerges from repeated next-token decisions. That framing explains capabilities, limits and costs.
What you will learn
- Define an LLM as a next-token estimator
- Explain why generation = repeated sampling
- Connect model choices to capability vs cost
What is this?
A large language model (LLM) is a neural network trained to predict the next token given all prior tokens. It outputs a probability distribution over likely continuations. To generate, you sample from that distribution and repeat. Everything you see — answers, summaries, agents — is many consecutive next-token draws, guided by the prompt and sampling settings.
Why do you need it?
- Every behavior stems from one objective — next-token prediction.
- It explains why models fabricate, stall or improvise: they optimize plausibility, not truth.
- It explains costs: generation is N sequential token predictions, each a model forward pass.
- It explains why context, system prompts and sampling reshape outputs.
Mental model
Autocomplete on steroids
Language models
Your phone autocompletes one word at a time from what you typed. An LLM does that at massive scale with a model that internalized enormous text patterns, so the next-word guesses are often paragraphs you actually wanted.
How it works
- Text is tokenized into chunks; each token has an ID.
- Forward pass produces a probability distribution over the whole vocabulary for the next token.
- Generation samples that distribution (greedy, top-p, temperature variants); the chosen token is fed back in.
- Repeat — this loop is all generation ever is.
- Scale + data = the emergent capabilities people call 'reasoning'.
Generation is a loop.
Examples
Top-k token sampling in spirit
import math, random class ToyModel: def __init__(self): self.vocab = ["hello", " world", " how", " are", " you"] def predict_next(self, context): # simulated probabilities over the vocab return [0.05, 0.55, 0.2, 0.15, 0.05] random.seed(4)m = ToyModel()tok = "LINE-0"for _ in range(4): probs = m.predict_next(tok) choice = random.choices(m.vocab, weights=probs)[0] print(choice, end="") tok += choiceA model that only returns a distribution + a sampler; that's the entire architecture of decoding from an LLM.
In production
- A chat completion is exactly this loop with the assistant-prompt in context.
- Retrieval-augmented prompts simply add tokens before generation starts.
- Cost models must multiply per-call tokens by per-token price, driven by next-token passes.
Best practices
- Approach LLMs as probabilistic tools, never oracles.
- Set sampling (temperature/top-p) per task; not all tasks want creativity.
- Design prompts that exploit knowledge of next-token behavior.
Common mistakes
- Expecting deterministic answers from generation without controls.
- Measuring latency without counting generation tokens.
- Boiling 'capability' down to model name size and ignoring context/tools.
Common issues
Model repeats the same sentence foreverHigh temperature + low-p under determinism or a degenerate sampling loop
Lower temperature, apply repetition penalty, cap max tokens.
Empty or early stop responsesStop-token scaffolding triggers on matched content
Check stop sequences, and validate output after the loop.
Debugging tips
- Repetitive output? Lower temperature or raise top-p filter accordingly.
- Fabrication increases as confidence drops — think fail-safe output validation.
- Same prompt, different answers — that's sampling, not a bug.
Interview checkpoint
Hands-on lab
- Model the 'predict next token' task with a tiny corpus
- Score candidate continuations deterministically
- Relate n-gram prediction to what an LLM does
Predict the Next Token, Statistically
25 min · beginner
Quick quiz
What is the fundamental operation of generation?
An LLM guarantees its outputs are factually correct.