Evaluation
Beginner 11 min read Evaluation

Evaluation Overview

Quality you can measure

Evaluation is how you know changes to prompts, models, or systems make things better — not just different. Offline tests, live metrics, and iterative loops turn model quality from vibes into engineering.

evaluationofflineonlinemetricsregressionquality
Prerequisites: Prompt Engineering
Before we start

What you will learn

  • Define eval layers: offline, online, and in-loop
  • Run evals before every change
  • Turn findings into regression tests
Introduction

What is this?

Evaluation is the practice of measuring model-system quality across defined dimensions using structured data and repeatable runs. Three layers: offline evals (golden sets in CI), online metrics (live traffic health), and in-loop checks (runtime gates). Each layer answers a question BREADTH before changes go out.

Motivation

Why do you need it?

  • You can't improve what you can't measure.
  • Prompts/models drift; evals catch regressions.
  • Evals replace anecdotal 'it looked better' with signals.
  • Graded, repeatable runs let the whole team iterate honestly.
Think about it

Mental model

Imagine

A doctor's vital signs

Think of

Eval layers

You don't assess health by gut feel: blood work (offline), resting vitals (online), bedside checks (in-loop). Each layer has its own cadence and purpose; the patient (system) gets worse before you feel it without all three.

Mechanics

How it works

  • Choose dimensions that matter for your product.
  • Build a golden dataset per dimension.
  • Run evals offline in CI on every change.
  • Track online health metrics on production traffic.
  • Roll findings into the golden set and iterate.

Three eval layers.

Offline
Online
In-loop
Iterate
Real world

In production

  • CI blocks prompt/model merges that regress goldens.
  • Dashboards watch answer quality on live traffic.
  • Runtime guards trigger fallbacks on degraded output.
Guidelines

Best practices

  • Eval before every prompt or model change.
  • Start with 20-50 high-value goldens, grow over time.
  • Separate automated metrics from human review.
  • Make evals deterministic (fixed seeds, mocks).
Watch out

Common mistakes

  • No baseline: changes judged by vibes.
  • Single 'good essay' test for a system that needs 10 dims.
  • Evals that run nowhere (docs, not loops).
Root cause analysis

Common issues

Error
Evals not used in workflow
Root cause

Not wired into CI/review

Fix

Run on merge; gate required scores.

Error
Golden set goes stale
Root cause

No refresh cadence

Fix

Harvest live failures every release into goldens.

When things break

Debugging tips

  • Evals pass, prod regresses → dataset drifted from traffic; harvest real cases.
  • Scores fluctuate run-to-run → nondeterministic model calls; seed/mock.
  • Too many failures to act → prioritize by severity and frequency.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Offline evals run…

2

Eyeballing two outputs is a substitute for evals.

Go deeper

Official documentation

Your AI quality rater (Anthropic)
docs
Evaluation guide (OpenAI)
docs