Evaluation
Advanced 12 min read Evaluation

Eval in CI/CD

Where evaluation becomes gating

Putting evals in the pipeline makes them part of engineering: run on merge, gate on deltas, and report to the team. CI/CD turns evaluation from a report into a decision.

cicdgatemergepipelinebudgetcache
Prerequisites: Offline Evals
Before we start

What you will learn

  • Wire evals into merge/pipeline steps
  • Gate on deltas with budgets
  • Report eval results to the team
Introduction

What is this?

Eval-in-CI/CD runs the suite automatically on every candidate change, compares against the stored baseline, and gates the merge on deltas. Budgets (token cost, time) make evals affordable in every pipeline; caching and staged suites (smoke → full) keep velocity while holding the line.

Motivation

Why do you need it?

  • Automated gates enforce the bar without meeting debates.
  • Blocker-budget deltas catch regressions before prod.
  • Caching and staging make heavy evals cheap enough for CI.
  • Reported results make eval a shared discipline.
Think about it

Mental model

Imagine

The factory gate check

Think of

Eval gates

Finished parts pass a quality gate before shipping — spot scale (smoke), then full tolerance profile (full suite). Gated, with a fast sort being caught early. Your pipeline's eval steps are that gate: cheap triage, honest full check, decision at the door.

Mechanics

How it works

  • Stage suites: smoke (fast) on push, full on merge/PR.
  • Compare current scores vs stored baseline.
  • Gate on deltas: block when below budget.
  • Cache heavy fixtures and offline results.
  • Post results to PR/chat for the team.

Pipeline gates.

Push
PR
Gate
Merge
Report
Real world

In production

  • Prompt changes merge only when golden pass rate holds.
  • Model upgrades stage through a fuller suite in release pipeline.
  • Guardrail rule edits gate on safety-suite deltas.
Guidelines

Best practices

  • Baseline stored per main branch commit.
  • Gate deltas, not absolute scores (fair for evolution).
  • Smoke first, full second; skip hours on safe changes.
  • Cache deterministic results; evals are repeatable by design.
Watch out

Common mistakes

  • Evals run but nobody gates — decoration.
  • Full suite on every push (slow, annoying, bypassed).
  • No baseline: every run is compared to nothing.
Root cause analysis

Common issues

Error
Evals block everything
Root cause

Delta budget inverted or stale baseline

Fix

Reset baseline on intended change; widen budget deliberately.

Error
Pipeline skips cheaps
Root cause

Eval too slow/costly

Fix

Cache + stage; run full suite on merge only.

When things break

Debugging tips

  • Pipeline too slow → stage suites; cache fixtures.
  • Gate flaky → determinism broke; fix seeds/mocks.
  • Regressions merged anyway → gate threshold bypassed; enforce.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

CI gates should compare eval scores to…

2

Running the full suite on every push is ideal.

Go deeper

Official documentation

LLM evals in CI (LangSmith)
docs