Prompt Regression Testing

• Build a golden prompt test suite
• Automate it in CI
• Gate model and prompt changes on suite results

• Prompts are user-facing; un-testled prompts rot quietly.
• Model upgrades silently detune prompts — the suite is the tripwire.
• Regression is generative: one bad example, reproduced everywhere.
• Fast, cheap, deterministic-enough — a handful of assertions beats a review argument.

You don't smell-test the kitchen each morning; a detector checks continuously and loudly when something changes. Model upgrades are like swapping your oven — the same recipe may now burn, and the detector (suite) tells you before dinner is ruined.

• Collect 20-50 golden inputs: happy paths, edge cases, safety probes.
• Write assertions: output parses, required fields present, banned phrases absent, refusal when appropriate.
• Pin stable decode (low temp, seed) to reduce flake.
• Run in CI on every prompt diff and model bump.
• Treat failures as code failures: block the merge.

• CI fails merges when a prompt stops producing valid JSON.
• Safety probes assert refusals for banned requests after every model bump.
• Golden format tests catch output-schema drifts before prod.

• Write assertions on STRUCTURE and DO/DO-NOT rules, not exact text.
• Mix normal inputs with adversarial probes.
• Re-run on model upgrades with a publish gate.
• Store fixtures with the prompt, readable by humans.

• Exact-match assertions on stochastic output (permanently red suite).
• Testing only happy-path formatting.
• Forgetting refusal/edge cases, so shipments pass on fluff.

• Flaky test → decouple from random decode; set seed/low temp, or require relaxed contain.
• Suite green but users unhappy → golden set unrepresentative; enrich from traffic.
• Model bump kills 3 tests → re-tune those prompts before shipping.