LLM Fundamentals
Intermediate 11 min read LLM Fundamentals

Temperature & Sampling

Turning probabilities into choices

Sampling settings shape the probability distribution — temperature flattens or sharpens it, top-p and top-k cut the tail. Deterministic tasks want low variance; creative tasks want room to roam.

temperaturetop-ptop-ksamplingnucleus samplingdeterminismcreativity
Prerequisites: Context Windows & Limits
Before we start

What you will learn

  • Explain temperature and top-p/top-k behavior
  • Choose settings per task type
  • Implement basic sampling in a few lines
Introduction

What is this?

After the model outputs a probability distribution over tokens, sampling converts it into a concrete token. Temperature scales logits (higher = flatter/random, lower = sharper/deterministic); top-p keeps the minimal set of tokens covering probability p; top-k keeps the k highest. All three tune the same dial from 'safest' to 'wildest'.

Motivation

Why do you need it?

  • Factual/code tasks collapse at high temperature — plausible-but-wrong takes over.
  • Creative writing and brainstorming collapse at zero temperature — endless repeats.
  • Top-p prunes absurd tails while keeping some freedom (vs naked temperature).
  • Determinism is expensive; when you need reproducibility, control sampling explicitly.
Think about it

Mental model

Imagine

A dimmer switch on the imagination

Think of

Sampling settings

Lowest dim = press the most-probable word every time (flat, safe prose). Highest dim = every word from a fevered turn (chaos). Top-p is a 'reasonable word filter' that keeps dimming meaningful but refuses nonsense.

Mechanics

How it works

  • Temperature: divide logits by T, then softmax. T=0 → argmax (greedy); T>1 → increasingly uniform.
  • Top-p: sort tokens by probability descending; keep the set whose cumulative probability reaches p; renormalize.
  • Top-k: keep only the k highest-probability tokens; renormalize.
  • T0 + top-p combined = quality-focused but not brittle.
  • Defaults from most platforms are the sweet spot until a task tells you otherwise.

Sampling choices on one distribution.

Logits
Scale
Filter
Softmax
Sample
Hands-on

Examples

Temperature math in plain Python

Temperature math in plain Python
import math, random
 
logits = [0.1, 2.0, 0.9, 0.4]
 
def softmax(v):
    e = [math.exp(x - max(v)) for x in v]
    s = sum(e)
    return [x/s for x in e]
 
print('T=0.5 (sharp) :', softmax([x/0.5 for x in logits]))
print('T=1.0 (normal):', softmax([x/1.0 for x in logits]))
print('T=2.0 (flat)  :', softmax([x/2.0 for x in logits]))

Higher temperature compresses differences until the bag equals chance; lower temperature sharpens the leader.

Real world

In production

  • Code generation uses low temperature (near 0) — determinism beats creativity.
  • Summarization prefers low temperature for consistency of facts.
  • Naming/product brainstorming runs high temperature + top-p tail filtering.
  • A/B loggers capture parameters so sample behavior isn't mysterious later.
Guidelines

Best practices

  • Pick settings per request type, not globally.
  • Never tune blindly; run the same prompts at a couple of settings on a small eval set.
  • Document parameters in production logs.
  • For reproducibility, seed + low temperature, and acknowledge residual variance.
Watch out

Common mistakes

  • Zero temperature as 'guaranteed determinism' for all tasks — it still varies by batch/knightess and platform.
  • Temperature-only with no top-p invites rare nonsensical tokens.
  • Set high temperature on accounting-style writing and paying for it later.
Root cause analysis

Common issues

Error
Repetitive degenerate output at low temperature
Root cause

Greedy decoding circling on a local pattern

Fix

Slight temperature (0.3) + top-p, add repetition penalty if available.

Error
Same prompt, two answers not matching
Root cause

Sampling implies variance; no seed used

Fix

Use seed/reproducible params for tests; log params; accept variance in prod.

When things break

Debugging tips

  • Creative tasks output bland repeats → raise temperature/top-p with discrimination.
  • Factual tasks invent creative-look info → drop temperature toward 0..0.2.
  • Outputs vary wildly at the same parameters across platforms → log the effective model + params.
Interviews

Interview checkpoint

Practice

Hands-on lab

Temperature Shapes the Sampling Distribution
25 min · intermediate
In progress
Objectives
  • Implement softmax with a temperature parameter
  • Show that lower temperature sharpens and higher flattens
  • Compare greedy picks and probability mass across temperatures

Temperature Shapes the Sampling Distribution

25 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which setting is the closest thing to deterministic decoding?

2

Top-p caps the cumulative probability mass of candidate tokens.

Go deeper

Official documentation

OpenAI text generation sampling params
docs