Guardrails
Advanced 12 min read Guardrails

The Sentry Pattern

A watcher for the watchers

The sentry pattern uses a second model to watch the first: an interloper that detects policy violations in output or drift in input, at a lower cost. Watchers who watch the watchers keep large systems legible.

sentrywatchdogsecond modelmonitoringshadowruntime check
Prerequisites: Guardrails Overview
Before we start

What you will learn

  • Wire a sentry model around a primary model
  • Bound sentry cost and false alarms
  • Escalate sentry hits appropriately
Introduction

What is this?

A sentry is a second, usually smaller/cheaper model that checks the primary's output (or input) against policy in the request path or off to the side. It catches drift and violations that prompts miss — without the sentry seeing the whole application state. Hits escalate to full validation or human review.

Motivation

Why do you need it?

  • A second set of eyes catches prompt-failure classes.
  • Small sentries are cheap enough to run every turn.
  • Sentry hits become the curated input to eval sets.
  • Observation is arguably enough: watch, then act.
Think about it

Mental model

Imagine

A manager spot-auditing their team's calls

Think of

The sentry

The manager (sentry) listens to a sample of calls, scores them against the script (policy), and escalates only the outliers. Cost: listening is cheaper than re-doing every call. Team still owns delivery; manager owns the exceptions.

Mechanics

How it works

  • Define what the sentry checks (policy dimensions).
  • Route a copy of output (or a sample) to the sentry model.
  • Score against rubric; flag below-threshold.
  • Escalate: re-validate with a stronger check or human.
  • Log hits; harvest into eval cases.

Sentry observation.

Primary
Sentry
Score
Pass
Escalate
Real world

In production

  • Agents route each tool result through a sentry for policy fit.
  • Support replies get sentry-scored before send.
  • Sentry flags in streaming feeds trigger moderation escalation.
Guidelines

Best practices

  • Keep the sentry independent of the primary's prompt (no shared bias).
  • Score cheaply; escalate rarely; human-review the flagged set.
  • Feed sentry hits into the golden suite as new cases.
  • Monitor sentry agreement with primary over time.
Watch out

Common mistakes

  • Sentry and primary share system prompts (blind spot double).
  • Sentry flags everything → threshold useless.
  • No escalation path — flags accumulate but nobody reviews.
Root cause analysis

Common issues

Error
Duplicate errors pass both
Root cause

Shared blind spots

Fix

Different model, rubric, or random prompt context for the sentry.

Error
Sentry cost balloons
Root cause

Full-score on everything

Fix

Score on samples and escalation-by-roulette over every turn.

When things break

Debugging tips

  • Sentry silent → check it's actually scoring; sample audit.
  • Sentry noisy → threshold drift; recalibrate on recent hits.
  • Sentry agrees with primary but both wrong → shared bias; vary perspective.
Interviews

Interview checkpoint

Practice

Hands-on lab

Score Outputs with a Mini Judge
35 min · advanced
In progress
Objectives
  • Run a small scoring function as a sentry
  • Implement threshold escalation
  • Feed hits into a golden queue

Score Outputs with a Mini Judge

35 min · advanced

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A sentry differs from the primary in that it…

2

Sentry hits belong in the eval suite.

Go deeper

Official documentation

Self-correct / guardrail patterns (Anthropic)
docs