Python
Intermediate 13 min read Python

Testing Python for AI

Determinism first, tokens later

AI code needs determinism: tests on parsing, chunking, prompting and tool dispatch catch regressions before they cost tokens. Pytest, fixtures and golden data make your pipeline auditable.

pytestfixturesgolden testsmockingdeterminismCI
Prerequisites: Package or structures
Before we start

What you will learn

  • Write pytest tests for pure functions like chunkers and parsers
  • Use fixtures to reuse fixtures and mock external model calls
  • Build golden tests that freeze output so regressions are instant
Introduction

What is this?

Testing AI code means asserting on what is deterministic — parsing, chunking, schema validation, prompt assembly, retrieval ranking — and mocking what isn't (live model calls). Pytest gives you fixtures, parametrization and plugins; golden tests freeze a known-good output so a change that breaks chunk boundaries or prompt format is caught in CI, not after paying token costs in production.

Motivation

Why do you need it?

  • A chunker regression silently ruins retrieval quality; a golden test makes it a red CI build.
  • Every debug cycle that hits a live API costs latency, tokens and nondeterminism.
  • Testing prompt/tool glue turns 'it mostly works' into 'it provably works'.
  • Graded on-call confidence: a failing test at 2am beats a failing trace at 3am.
Think about it

Mental model

Imagine

A chef tasting before serving

Think of

Golden tests for an AI pipeline

The chef knows the dish by taste — a tiny deviation is noticed instantly. Golden tests are that calibrated taste: a known-good reference you compare every change against.

Mechanics

How it works

  • Tests are functions named `test_*` under a `tests/` dir; pytest collects and runs them.
  • Fixtures provide reusable setup — a sample corpus, a fake client — per test.
  • `responses`/`respx` or monkeypatching intercept HTTP so model calls never leave the test.
  • Golden tests run a function on fixed input and assert exact output or a hash.
  • CI runs linters, mypy and pytest on every PR so regressions block merge.

The test pyramid for an AI service.

Unit
Golden
Integration
E2E
Hands-on

Examples

A golden test for a chunker

A golden test for a chunker
def chunk(text, size):
    words = text.split()
    return [" ".join(words[i:i+size]) for i in range(0, len(words), size)]
 
def test_chunk_golden():
    text = "alpha beta gamma delta epsilon"
    assert chunk(text, 2) == ["alpha beta", "gamma delta", "epsilon"]

A pure function with a frozen expected output is a golden test: any chunk-boundary change fails the build immediately.

Real world

In production

  • A RAG service tests chunk sizes, overlap and ID stability against golden corpora, so a library upgrade can't silently change boundaries.
  • An agent project mocks the LLM client in tests, asserting the dispatched tool calls and their argument shapes.
  • A guardrail library parametrizes a suite of injection payloads against its filters, with expected block/allow triages.
Guidelines

Best practices

  • Keep tests on pure functions; mock external I/O at the boundary.
  • Seed golden data as files in the repo, not constants regenerated at runtime.
  • Mark slow, e2e or keyed tests with pytest markers and exclude from CI unless requested.
  • Run lint, type-check and tests in CI so every change is gated identically.
Watch out

Common mistakes

  • Testing only the happy path and shipping parsers that blow up on one edge row.
  • Golden tests that regenerate their own expectation, so they always pass.
  • Hitting live API keys in CI, making runs slow, flaky and expensive.
  • No tests around prompt/tool glue — the layer where production bugs actually live.
Root cause analysis

Common issues

Error
Golden test fails after an SDK update
Root cause

A dependency changed tokenization or serialization behavior

Fix

Review the changelog, update the golden expectation deliberately and pin the version.

Error
Tests pass locally, fail in CI
Root cause

Different dependency resolution or missing env config

Fix

Install from the same lockfile and make CI use the same env vars.

Error
Mocked model returns canned text but a test still hits the network
Root cause

The code path uses a different client than the one mocked

Fix

Inject the client into the function so tests can substitute the fake cleanly.

When things break

Debugging tips

  • A golden test failing after a dep upgrade: diff actual vs expected and read the dependency changelog.
  • Flaky tests usually mean nondeterminism leaked in — look for unsorted retrieval or live HTTP.
  • When a test passes locally but fails in CI, compare dependency versions and env vars.
  • For time-based logic, inject the clock (freezegun) rather than patching sleep.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Golden tests should generate their own expected output at runtime.

2

Where should live LLM API calls happen in a test suite?

Go deeper

Official documentation

pytest documentation
docs
Hypothesis property testing
docs