Evaluation
Advanced 13 min read Evaluation

LLM as Judge

Scaling grading with models

Use a model to grade models: rubric-driven, fast, scalable evaluation where human annotation can't scale. Judges need calibration, position-bias control, and human spot-checks to stay trustworthy.

llm judgerubriccalibrationposition biasself-checkscalable grading
Prerequisites: Eval Metrics
Before we start

What you will learn

  • Author judge rubrics
  • Control judge biases
  • Calibrate judge against human grades
Introduction

What is this?

An LLM-as-judge grades answers with a rubric rather than matching strings — fast and scalable across big eval sets. Trustworthiness requires control: explicitly separate instructions from the judged content, avoid position bias (vary order in pair comparisons), and periodically calibrate the judge against human grades.

Motivation

Why do you need it?

  • Human review doesn't scale to thousands of variants.
  • Judges handle open-ended multi-dimension quality cheaply.
  • Rubrics give judges (and humans) a shared standard.
  • Calibrated judges approach agreement with careful human grading.
Think about it

Mental model

Imagine

A trained examiner with a rubric

Think of

LLM judge

Examiners don't grade by vibes; they use a rubric (criteria + anchors), read blind where possible, and their grading is audited against a panel (calibration). An untrained examiner is noise; a trained, calibrated one is repeatable. Your judge model needs the same discipline.

Mechanics

How it works

  • Write a rubric: dimensions, anchors, pass rules.
  • Prompt the judge with role + rubric + item, nothing else mixed in.
  • Randomize presentation order for pair grades.
  • Run on a sample; compare against human grades.
  • Adjust rubric/threshold until judge-human agreement is strong.

Judge loop.

Rubric
Judge
Calibrate
Audit
Scale
Real world

In production

  • Large prompt sweeps graded quickly through judges.
  • RAG evals use judges for coherence+ groundedness.
  • Support-answer evals score tone+resolution with rubric.
Guidelines

Best practices

  • Separate the artifact from instructions (no concatenated style bleed).
  • Randomize order in A/B judge comparisons.
  • Calibrate on samples where humans graded first.
  • Re-calibrate judges when models change.
Watch out

Common mistakes

  • Judge 'agrees' because it saw the answer before grading.
  • Position bias: option A always wins in pair grading.
  • No human audit — the judge rot goes unnoticed.
Root cause analysis

Common issues

Error
Judge disagrees with humans
Root cause

Poor rubric or judge model

Fix

Re-author anchors; try a stronger judge; sample humans.

Error
Judge suspects the graded answer is its own
Root cause

Leakage in prompt

Fix

Clean separation; randomize; don't include reference drafts.

When things break

Debugging tips

  • Judge favors style over logic → rubric calls out correctness explicitly.
  • Scores cluster at ceiling → anchors too generous; re-anchor.
  • Judge-model change shifts grades → re-calibrate and re-run baseline evals.
Interviews

Interview checkpoint

Practice

Hands-on lab

Simulate an LLM Judge
30 min · intermediate
In progress
Objectives
  • Score outputs against a rubric with attributes
  • Apply the judge to a response batch
  • Aggregate scores and flag weak items

Simulate an LLM Judge

30 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Position bias in pair grading is controlled by…

2

Judge models need periodic human calibration.

Go deeper

Official documentation

LLM-as-a-judge (OpenAI)
docs