CI/CD for LLMs
When the artifact is a prompt and a weights file
LLMs break classic CI/CD assumptions: prompts, not just weights, are the artifact; evaluation needs evalsets and judges; and cost/latency are checks. CI/CD for LLMs version prompts, weights and evals, and gates on honest outcomes.
What you will learn
- Version the prompt, weights and evalset together
- Build evaluation into CI with judges and baselines
- Gate LLM releases on cost, latency and safety, not just accuracy
What is this?
CI/CD for LLMs adapts the loop to models whose behavior is jointly defined by weights and prompts: it versions prompt + weights + config + evalsets as one deployable unit, runs evaluation with an evalset and judge (human or LLM-as-judge), and gates release on accuracy AND cost, latency and safety.
Why do you need it?
- A prompt edit can flip behavior as much as a new checkpoint — both need versioning.
- LLM output has no single ground truth; objective evalsets and judges provide the metric.
- Cost and latency swing with context and configuration, worth gating like quality.
- Model upgrades silently change responses — regression evals catch 'better in general, worse for our users'.
Mental model
A chef testing a dish on a panel before putting it on the menu
LLM CI/CD
You can't taste the soup by measuring the pan. The chef checks a defined tasting protocol (evalset), compares to the previous dish (baseline), and weighs price vs quality (cost/latency) before the menu change (release).
How it works
- A change touches prompt, weights, config or evalset → triggers LLM CD.
- The evalset runs deterministic samples with expected behaviors.
- A judge evaluates open responses; human spot-checks keep the judge honest.
- Cost and latency are measured per request and checked against budgets.
- Results compare to the deployed baseline; regression blocks the rollout.
- A safe rollout deploys to a shadow/canary and records real outcomes.
The LLM release loop.
In production
- A chatbot team versions prompt + model + evalset in one repo; each PR runs a 300-item eval.
- An upgrade to a new backbone is evaluated against cached previous answers via LLM judge.
- A cost gate blocks a prompt change that triples token spend per conversation.
Best practices
- Treat the prompt as a reviewed artifact with its own tests.
- Keep a golden evalset, refreshed against real user traffic, and a held-out variant for final gates.
- Use varied LLM judges plus human spot-checks to avoid judge self-bias.
- Measure cost/latency as release criteria, not afterthoughts.
Common mistakes
- Gating only on 'felt better' demos — no evalset, no baseline, no judge.
- Letting the same model grade itself exclusively (judge bias).
- Shipping a prompt change with no diff on token cost.
- Model upgrade that changed behavior silently, never run against the old outputs.
Debugging tips
- Evals fail after a model upgrade — diff old vs new outputs; judge disagreements are where the real regressions hide.
- Cost spike not in the eval — measure tokens on real-shaped requests, not the tiny eval fixture.
- 'Judge loves it, users hate it' — your evalset drifted from user behavior; rebuild from real traffic.
Interview checkpoint
Hands-on lab
- Version a prompt alongside a config and evalset
- Run a deterministic evalset with a simple judge
- Gate release on score, cost and latency budgets
Gate an LLM Prompt Change
40 min · advanced
Quick quiz
In LLM CI/CD, what should the release gate evaluate beyond accuracy?
An LLM upgrade should be compared against the previously deployed answers before rollout.