Generative AI
Intermediate 15 min read Generative AI

Inference & Decoding Strategies

What the model says: sampling, search, and their knobs

Greedy decoding always picks the argmax token; sampling with temperature, top-k and top-p trades determinism for diversity. Beam search and length penalties tune long-form quality. Decoding choices are product decisions.

greedybeam searchtemperaturetop-ktop-pnucleus samplingrepetition penaltyKV cache
Prerequisites: Language Model Training
Before we start

What you will learn

  • Explain greedy, beam search and sampling and their differences
  • Tune temperature, top-k, top-p for a use case
  • Explain KV-cache decoding and why latency is proportional to output length
Introduction

What is this?

At each step the LM outputs a probability over the vocabulary; the decoding strategy decides which token becomes the output. Greedy takes the argmax every time — deterministic but formulaic. Beam search keeps the top-k candidate sequences, trading compute for coherence. Sampling draws from the (possibly re-weighted) distribution; temperature sharpens or flattens it, top-k keeps the k most likely tokens, top-p keeps the smallest set covering probability mass p. Autoregressive inference with a KV-cache makes each new token O(1) rather than O(sequence).

Motivation

Why do you need it?

  • Repetition and blandness come from greedy/argmax — the mode is boring.
  • Diversity (top-p + temperature) is what makes creative writing and code comments human.
  • Determinism matters for tests, parsing and latency-sensitive decision paths.
  • Understanding decoding = predictable latency, cost and output quality per product.
Think about it

Mental model

Imagine

Choosing restaurant dishes for a perfect dinner

Think of

Decoding strategies

Greedy = always pick your favorite dish again. Beam = consider the top 3 ten-course menus and pick the best whole menu. Sampling = roll a die weighted by how much you like each dish, with temperature as your adventurousness slider.

Mechanics

How it works

  • greedy: out_t = argmax P(vocab | prefix) — deterministic, fastest, least diverse.
  • beam search (k): track the k most probable full sequences, extend each, keep best k, return highest-scoring — coherent but costly and often formal.
  • temperature: divide logits by T before softmax; T→0 is greedy, T≈1 is raw, T>1 flattens to randomness.
  • top-k: restrict sampling to top-k logits then re-normalize; top-p: smallest set containing cumulative probability p.
  • KV cache: cache the K/V tensors of all past tokens so each new token only recomputes its own attention — the reason per-token latency flattens.

A distribution, after temperature and truncation filters.

Logits
Temperature
top-k / top-p
Sample / argmax
Append + KV cache
Real world

In production

  • Deterministic extraction/parsing: temperature 0 or low with strict formats.
  • Creative drafting: temperature 0.8-1.0 with top-p 0.9.
  • Code assistance: top-p sampling with repetition penalty for fewer loops-forever bugs.
Guidelines

Best practices

  • Use temperature + top-p together as one dial (nucleus sampling); don't tune in isolation.
  • For JSON/structured output, prefer temperature 0 plus schema validation over hoping sampling complies.
  • Add a repetition/coverage penalty for open-ended long-form; keep it small (1.0-1.15).
  • Seed the RNG per request so you can replay and debug any sampled output.
Watch out

Common mistakes

  • Blasting temperature 2.0 to 'fix boredom' and getting gibberish.
  • Using beam search for creative text where repetition creeps into sentences.
  • Ignoring that greedy can lock into a loop; sampling breaks it.
  • Forgetting the KV cache and measuring O(n²) latency that flash-decoding then contradicts.
Root cause analysis

Common issues

Error
Repetitive loops in long outputs
Root cause

Greedy/beam mode collapsed onto a template

Fix

Switch to top-p (0.9) sampling, add a mild repetition penalty, or raise temperature.

Error
Structured output frequently invalid JSON
Root cause

Sampling explores token tails that break grammar

Fix

Use temperature 0 + JSON mode/schema, or validated decoding.

Error
Answers vary run to run in production
Root cause

Sampling active on a task that should be deterministic

Fix

Set temperature 0 for the task, fix seed, and log the config for audit.

When things break

Debugging tips

  • Log temperature/top-p/top-k naive values with every completion so you can replay failures.
  • Probe with one prompt across seeds: high variance = sampling settings or prompt underspecification.
  • Watch output-length histograms — a spike near max_tokens usually means truncation, not natural endings.
  • Compare greedy vs sampled on your eval set; if greedy wins, sampling probably hurts your task.
Interviews

Interview checkpoint

Practice

Hands-on lab

Compare Decoding Strategies
30 min · intermediate
In progress
Objectives
  • Simulate greedy, temperature and top-p sampling on a toy distribution
  • Measure diversity and determinism across strategies
  • See why temperature/top-p modulate output tails

Compare Decoding Strategies

30 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Lowering temperature toward 0 makes sampling:

2

Beam search guarantees the most creative, diverse output.

Go deeper

Official documentation

Hugging Face generation guide
docs