Evaluation
Advanced 13 min read Evaluation

Offline Evaluations

Comparable runs, trusted diffs

Offline evals are deterministic, repeatable runs of the system against golden data — the backbone of trusted iteration. Designing the run (seed, mocks, versions) makes results comparable across code changes.

offline evaldeterminismrerunfixturescomparisonbaseline
Prerequisites: LLM as Judge
Before we start

What you will learn

  • Structure reproducible eval runs
  • Mock external systems for determinism
  • Interpret deltas between versions
Introduction

What is this?

Offline evaluation runs your system against a fixed golden set in a controlled environment: fixed model params, seeded sampling, mocked external tools — so score deltas mean the code or prompt changed, not the weather. Locking versions and invoking metrics identically makes every run comparable to every baseline.

Motivation

Why do you need it?

  • Comparable runs turn regressions into binary facts.
  • Mocks remove flaky external sources of failure.
  • Fixed seeds make sampling-effects visible.
  • Library/protocol pinning gives long-run comparability.
Think about it

Mental model

Imagine

A lab with controlled conditions

Think of

Offline evals

You wouldn't compare two drug batches in different weather with different scales. Labs control temperature, instruments, and protocols. Offline evals control seeds, mocks, and versions — so the only variable left is the change you're testing.

Mechanics

How it works

  • Freeze the golden set and eval parameters.
  • Mock tools/APIs that change answer fate.
  • Pin models, SDKs, and prompts (as code).
  • Run, record raw outputs + scores.
  • Compare against baselines; emit a structured report.

Controlled run.

Lock
Run
Record
Diff
Gate
Real world

In production

  • Every prompt/model diff runs the full golden suite in CI.
  • A/B of prompt-variant sets compare offline before any traffic.
  • Release notes carry the eval delta as evidence.
Guidelines

Best practices

  • One harness, one set, one metric config per eval family.
  • Store outputs + scores, not just scores.
  • Pin everything that influences output.
  • Gate merges on deltas to baselines.
Watch out

Common mistakes

  • Comparisons across different seeds (phantom deltas).
  • Real-network calls in offline runs (flaky + slow).
  • No baseline snapshot — deltas are unreadable.
Root cause analysis

Common issues

Error
Offline differs from prod
Root cause

Mocks diverge from real systems

Fix

Periodically compare mock vs real; refresh fixtures.

Error
Results not reproducible
Root cause

Undeterministic calls

Fix

Seed sampling and pin versions end-to-end.

When things break

Debugging tips

  • Score wobble → check seed/mock/pin drift; rerun clean.
  • Delta unexplained → diff the recorded outputs, not just scores.
  • Suite times out → mocks missing; replace real calls.
Interviews

Interview checkpoint

Practice

Hands-on lab

Deterministic Golden-Suite Runner
40 min · advanced
In progress
Objectives
  • Lock a golden set + seeds
  • Run a reproducible eval suite
  • Diff results against a baseline

Deterministic Golden-Suite Runner

40 min · advanced

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Seeding makes offline runs…

2

Real network calls belong in offline eval runs.

Go deeper

Official documentation

LangSmith evaluation
docs