Data Handling
Intermediate 14 min read Data Handling

Data Validation for AI Pipelines

Contracts that catch the lies early

Every AI pipeline ingests data that lies: nulls where numbers should be, empty strings, schema drift, adversarial payloads. Validation contracts fail fast and make the pipeline trustworthy.

validationcontractsschemaassertionsdata qualitypydantic
Prerequisites: JSON Handling
Before we start

What you will learn

  • Validate inputs and outputs at every stage boundary
  • Define contracts that fail loudly with actionable messages
  • Add column/field-level checks to catch silent drift
Introduction

What is this?

Data validation means asserting that what enters and exits each pipeline stage matches an expected contract: types, required fields, value ranges, cardinality. Tools like pydantic, JSON Schema and assertion libraries turn malformed data into immediate, descriptive failures instead of downstream corruption. For AI, validation also guards model output against your own schemas and your inputs against adversarial payloads.

Motivation

Why do you need it?

  • A null where a length should be can poison re-ranking math with NaNs that look 'working'.
  • Empty strings embedded as 'content' create useless, memory-wasting vectors.
  • Schema drift from an upstream API should be an alert, not a silent behavior change.
  • Validation at boundaries makes root-causing trivial: the failure names the field and the stage.
Think about it

Mental model

Imagine

Airport security at every gate

Think of

Validation at each pipeline stage

Security checks at entry, not only at the final destination. If you only check at the end, a poisoned bag has already flown the whole route. Validation early confines damage to one stage.

Mechanics

How it works

  • Define contracts: for each record, the expected type, required fields, allowed values and reasonable ranges.
  • Validate at the ingest boundary (external API/file), mid-pipeline (stage output) and pre-model (prompt inputs).
  • Use pydantic/JSON Schema which coerce or reject; policy differs: reject on type, coerce on small deviations.
  • Add cardinality checks (row counts, unique keys) so 'silent-drop' failures become loud.
  • Alerts route the failure to the owning stage rather than a generic page.

Validation checkpoints across the pipeline.

Ingest
Extract/normalize
Chunk/embed
Model output
Serve
Real world

In production

  • An ingest rejects rows missing the natural key and alerts — before they land in the RAG index as duplicates.
  • A structured-output service validates model JSON against the requested schema and triggers a corrective retry on mismatch.
  • A daily corpus pipeline fails the run when chunk count deviates more than 10% from baseline, catching silent drops.
Guidelines

Best practices

  • Validate at stage boundaries, not everywhere — one contract per boundary.
  • Reject on type errors; coerce only consciously-defined deviations.
  • Make failure messages actionable: name the field, the record, and the stage.
  • Validate model output against the same schema the caller requested.
Watch out

Common mistakes

  • Only validating happy-path inputs, letting output drift persist for weeks.
  • Catching ValidationError and logging without re-raising, silently running on bad data.
  • Validating production but not the eval harness, making eval results non-predictive.
  • Accepting strings where enums should be, then surprise-breaking on a new value.
Root cause analysis

Common issues

Error
NaN leaked into chunk lengths
Root cause

A float field accepted nulls or non-numeric strings

Fix

Declare the field as float with a range and reject NaN/None explicitly.

Error
Pipeline passes but a table is empty
Root cause

Cardinality checks missing; filter quietly returned nothing

Fix

Assert expected row-count bands and fail on empty-at-all-cost stages.

Error
New enum value from upstream breaks dispatch
Root cause

Contract allowed any string instead of an enum

Fix

Enum the field and add explicit handling (or rejection) for new values.

When things break

Debugging tips

  • Trace a bad record's path backward: the first boundary to fail is also the first to blame.
  • Log a sample of rejected records (redacted) to spot bulk drift early.
  • When 'it worked before', diff the upstream schema changelog against your contract.
Interviews

Interview checkpoint

Practice

Hands-on lab

Contract-Gate Your Ingest
30 min · intermediate
In progress
Objectives
  • Validate incoming records against a pydantic-like contract
  • Reject bad rows and report why, without losing the rest
  • Add a cardinality guard that fails empty-but-green runs

Contract-Gate Your Ingest

30 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

What does pydantic give you beyond dict indexing?

2

Validation should only happen at the external ingest boundary.

Go deeper

Official documentation

pydantic docs
docs
Great Expectations
docs