Evaluation
Intermediate 12 min read Evaluation

Eval Metrics

From outputs to numbers

Metrics make quality measurable: exact-match and semantic similarity for factual tasks, rubric scores for reasoning, entailment for groundedness, and pass rates aggregated into one gated number.

metricsexact matchsemantic similarityrubricentailmentprecisionrecall
Prerequisites: Golden Tests
Before we start

What you will learn

  • Choose metrics per task type
  • Combine metrics into gate decisions
  • Avoid metric-gaming blind spots
Introduction

What is this?

Evaluation metrics convert outputs into comparable numbers. Exact match (code/classification), semantic similarity (paraphrase-tolerant), rubric scores (reasoning, multi-dimension), entailment/groundedness (does answer stay in sources), plus precision/recall over structured items. Aggregated pass rate becomes the gate; chosen per task, never one-size.

Motivation

Why do you need it?

  • Different tasks falsify differently — metrics must match.
  • Semantic metrics catch 'right meaning, different words'.
  • Groundedness metrics catch fabrication where prose looks fine.
  • A single aggregate makes gating decisions legible.
Think about it

Mental model

Imagine

Scoring in a decathlon

Think of

Metric choice

You don't score a 100m sprint by gymnastic form; each event has a metric suited to it. Aggregate medals come from summing the right per-event scores. Your eval suite is a decathlon: exact match for the sprints, rubrics for the leaps, entailment for the throws.

Mechanics

How it works

  • Classify the task: structured facts, prose, reasoning, extraction.
  • Pick the metric: exact/match/containment, similarity, rubric, entailment.
  • Define pass thresholds per dimension.
  • Aggregate (pass rate, per-dimension averages) for gates.
  • Track metric distribution, not just the mean.

Metric families.

Structured
Paraphrase
Reasoning
Grounded
Gate
Real world

In production

  • Extraction tasks gate on field-level F1.
  • Customer-support summaries gate on rubric + groundedness.
  • Codegen gate on tests passing + exact match of key symbols.
Guidelines

Best practices

  • Match the metric family to the failure you fear.
  • Use thresholds per dimension, not a single magic number.
  • Report distributions (percentiles) so outliers aren't hidden.
  • Pair automated metrics with spot human review.
Watch out

Common mistakes

  • Exact match on prose (paraphrase-punishing).
  • Rubric on factual ticks (subjectivity creep).
  • Gaming a single aggregate by overfitting goldens.
Root cause analysis

Common issues

Error
Metric contradicts human judgment
Root cause

Metric/task mismatch

Fix

Re-classify the task; pick the right metric family.

Error
Pass rate unstable
Root cause

Model nondeterminism

Fix

Seed/sample the evaluation n times; report confidence.

When things break

Debugging tips

  • Scores high but human sees junk → metric gaming blind spot; add a rubric/human layer.
  • Semantic similarity flags nonsense → threshold loose; tighten.
  • Groundedness misses → entailment model too weak; sample instead.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Groundedness measurement checks…

2

One aggregate number captures all quality.

Go deeper

Official documentation

Ragas metrics
docs