Prompt Engineering
Advanced 13 min read Prompt Engineering

Prompt Tuning & Iteration

Tuning prompts with a scoreboard

Iterating on prompts is a scientific loop: hypothesis → change → eval → decide. Variable-by-variable A/B and regression suites turn prompt tuning from artistic to measurable.

prompt tuningiterationA/Beval-drivenprompt optimization
Prerequisites: Prompt Testing
Before we start

What you will learn

  • Run an eval-driven prompt iteration loop
  • Isolate one variable per change
  • Decide ship vs revert by numbers
Introduction

What is this?

Prompt tuning is the process of improving a prompt measurably: run a golden eval, change one variable (phrasing, examples, format), re-evaluate, and keep changes that lift scores. Combined with versioning, this becomes the discipline that keeps prompt quality high across model upgrades.

Motivation

Why do you need it?

  • Eval-driven iteration compounds; vibes-driven 'tweaks' collect randomly.
  • One-variable-at-a-time keeps causality clean enough to learn from.
  • Prompt deltas are cheap to test and replay.
  • A regression suite amortizes every future model-upgrade decision.
Think about it

Mental model

Imagine

A kitchen recipe refined through tastings

Think of

Prompt tuning

You don't change salt, temperature and plating at once and guess why the dish changed. The tasting panel (eval) scores each single change; the recipe (prompt) versions up when scores rise. Perfection is dozens of small measured adjustments.

Mechanics

How it works

  • Detach a golden eval set with a clear metric.
  • Baseline: run the current prompt. Record number.
  • Change ONE variable; re-run. Compare.
  • Ship winning delta; log it; revert on regression.
  • Repeat vs a budget (changes per release).

The tuning loop.

Baseline
Hypothesis
Eval
Decide
Log
Real world

In production

  • Weekly tuning on sampled traffic for chat support prompts.
  • Model-upgrade gates rerun the suite to catch prompt detuning.
  • Feature A/B: the same prompt tuned per model version in prod config.
Guidelines

Best practices

  • Always baseline first; never 'improve' without a number.
  • Change one thing per experiment.
  • Keep a prompt changelog with metrics (before → after).
  • Re-check regressions on the FULL suite, not the happy path.
Watch out

Common mistakes

  • Tuning on a handful of eyeballed examples.
  • Multiple concurrent edits making attribution impossible.
  • Skipping the eval after the actual model upgrade.
  • Never reverting — clinging to a trend line of one experiment.
Root cause analysis

Common issues

Error
Eval set too small to measure
Root cause

High variance rundown

Fix

Enlarge golden set or aggregate multiple runs.

Error
Prompt tuned but prod still bad
Root cause

Golden set ≠ production distribution

Fix

Sample real traffic into the golden set.

When things break

Debugging tips

  • Prompt change 'feels' better but eval flat → revert; the product didn't move.
  • Regression but phrasing 'shouldn't matter' → it does; the suite caught it.
  • Model upgrade flipped scores → prompt was tuned on the old distribution; re-tune.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The rule for prompt edits is…

2

A prompt tuned for model A is automatically tuned for model B.

Go deeper

Official documentation

Prompt tuning loop (OpenAI)
docs