Production AI Architecture
Advanced 12 min read Production AI Architecture

CI/CD for LLM Apps

Ship the string and the model responsibly

LLM deploys ship prompts, configs, models and callable code — so CI/CD runs golden evals, prompt lint for the failure patterns, redaction checks, cost gates, and regression tests of tool logic, with visual diffing of prompt changes before release.

CI/CDgolden evalprompt lintcost gate
Prerequisites: Evaluation in Production
Before we start

What you will learn

  • Gate changes on evals and prompt lint
  • Automate peek-and-review for prompt diffs
  • Make cost and security check gates
Introduction

What is this?

LLM CI/CD treats prompts, config and model choices as deployable artifacts: a PR changes a prompt, so the pipeline runs golden evals, lints for injection and secret patterns, checks redaction, runs unit tests on tool/parse code, checks expected cost deltas, and produces a human-reviewable diff of exactly what changed in behavior.

Motivation

Why do you need it?

  • A one-line string change can flip production behavior — it deserves a review gate like any code.
  • Golden evals catch quality regressions a linter cannot.
  • Cost and safety are properties of the prompt/config, so they gate the same pipeline.
Think about it

Mental model

Imagine

LLM CI/CD

Think of

Sending a manuscript to reviewers

The editors (CI) check grammar and fact (evals/lint) and the author's board (human review) sees an annotated diff — one changed sentence might reshape the chapter.

Mechanics

How it works

  • Lint prompt files for injection probes, secret patterns, overlong inputs, and drift from template.
  • Run golden evals on the changed prompt/model: score correctness, safety, format.
  • Unit-test the deterministic glue: parsing, tool calls, JSON reshaping.
  • Diff evals-and-cost vs the last release; require humans to eyeball the behavioral delta on risky changes.

LLM pipeline gates.

PR
Lint
Eval
Cost
Ship
Real world

In production

  • A chat team's PR pipeline refuses prompt changes with regression on factuality evals.
  • An agent repo tests tool-parse edge cases in unit tests plus injection-suppression in goldens.
  • A financially-gated copilot blocks releases whose estimated cost delta exceeds 15%.
Guidelines

Best practices

  • Run evals consistently: same golden set, same judge config, normalizing scores to a baseline.
  • Make behavioral diffs human-checkable: render before/after paraphrasings on a sample set.
  • Keep golden sets evolving with real prod traffic samples.
  • Gate on gates: prompt lint and eval regressions fail the build, not just warn.
Watch out

Common mistakes

  • Evocative but non-deterministic evals — flaky gates, ignored gates.
  • Only code in the pipeline, string changes shipping with no review.
  • Cost estimated by guessing instead of measured token tables.
When things break

Debugging tips

  • Eval flaky → pin judge/params and normalize scores against a captured baseline.
  • Prompt diff 'looks identical' → render behavior on the golden set; phrasing shifts show there.
  • Cost gate whitelisting everything → compare estimated vs actual post-release spend monthly.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The new artifact type LLM CI/CD must gate is?

2

Golden evals alone make prompt lint unnecessary.

Go deeper

Official documentation

LangSmith evaluation
docs
Promptfoo
docs