Evaluation
Intermediate 12 min read Evaluation

Golden Tests

Fixed truth, repeatable verdicts

Golden tests are curated input→expected-output pairs with rubrics — the ground truth your system must keep meeting. Small, sharp, and versioned, they're the regression net every change runs against.

golden testsgolden setrubricregressionfixtures
Prerequisites: Evaluation Overview
Before we start

What you will learn

  • Author golden tests with rubrics
  • Balance coverage across dimensions
  • Keep goldens sharp and current
Introduction

What is this?

Golden tests pair an input with an expected output or rubric (dimensions graded 1-5, plus acceptance rules). They encode the bar your system must hold. A small, carefully chosen golden set covering each product dimension beats a huge vague one — and it grows by harvesting real failures.

Motivation

Why do you need it?

  • Fixed datasets make regressions visible across any change.
  • Rubrics let humans grade while automation checks facts.
  • Golden failures are precise, actionable diffs.
  • Versioned goldens travel with the codebase.
Think about it

Mental model

Imagine

Calibration weights on a lab scale

Think of

Golden tests

You don't trust a scale without known weights: place the 1kg weight, read the needles. Golden tests are the known weights of your output scale — fixed, certified, and re-checked on every change to the machine.

Mechanics

How it works

  • Pick inputs that stress each product dimension.
  • Define expected behavior: exact match, rubric, or constraints.
  • Automate the runnable parts; run in CI.
  • Grade, relax or add goldens as the product evolves.
  • Track pass/fail over time as regression signal.

Golden lifecycle.

Author
Run
Grade
Refresh
Gate
Real world

In production

  • Each prompt/model change gates on golden pass rate.
  • Dispute reviews use goldens as the shared baseline.
  • Live failures get copied into the golden set as new cases.
Guidelines

Best practices

  • Quality over quantity: cover dimensions, not volume.
  • Write rubrics before running, to avoid grade drift.
  • Version goldens with the product decision.
  • Automatically add confirmed live regressions to the set.
Watch out

Common mistakes

  • Fuzzy expectations ('good enough') with no rubric.
  • Goldens stolen from old prompts that no longer apply.
  • Grading 'totally fine' to protect merge velocity.
Root cause analysis

Common issues

Error
Golden set too big to maintain
Root cause

Volume culture

Fix

Trim to high-value dimension cases; archive the rest.

Error
Golden expectations stale
Root cause

Product moved

Fix

Review at release cadence; document expected changes.

When things break

Debugging tips

  • Golden fails after change → run the diff; likely a real regression, fix or update consciously.
  • Golden passes always → too lenient or not covering risk; tighten.
  • Tests too brittle → expect exact prose; prefer constraints/rubrics.
Interviews

Interview checkpoint

Practice

Hands-on lab

Golden-Test Harness
25 min · beginner
In progress
Objectives
  • Define a golden set of input to expected output
  • Score a completion against expected content
  • Summarize pass rate across the set

Golden-Test Harness

25 min · beginner

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A good golden test includes…

2

Golden sets should mainly grow, never prune.

Go deeper

Official documentation

Test-driven evaluation (Anthropic)
docs