CI for ML
Fast, meaningful checks before anything ships
Continuous Integration for ML runs fast, meaningful checks on commits: data schema, code quality, a minimal training smoke test and metric baselines. The goal is catching regressions in minutes, not at retraining time.
What you will learn
- Design a CI stage list that catches real ML regressions
- Add a smoke train that runs in minutes, not hours
- Assert metric baselines so silent degradation fails the build
What is this?
CI for ML is the automated gate that runs on every commit: check code quality, validate the data contract, run a scaled-down training smoke test and assert that produced metrics clear a champion baseline — all inside minutes so developers get fast, honest feedback.
Why do you need it?
- A broken schema or a new outlier profile can silently wreck a retrain weeks later.
- Fast feedback makes fixing cheap and keeps the loop tight.
- Baseline assertions turn 'model got worse' into a failing check, not a postmortem.
- Smoke trains verify the training path itself before expensive full runs.
Mental model
A restaurant's line check before service
ML CI
Chefs check one perfect plate of each dish (smoke test), verify ingredient labels (schema) and the pass has standards for what leaves (baselines). A wrong special doesn't go out just because the oven works.
How it works
- A commit triggers CI on the training repo.
- Static stage: lint, formatting, unit tests for data and eval helpers.
- Data stage: schema assertions against a pinned fixture dataset.
- Smoke stage: train the model on a tiny stratified subset for minutes.
- Assertions: eval code runs and metrics clear a configured baseline threshold.
- On success, CI tags the commit and updates the candidate's evidence bundle.
Fast-feedback ML CI stages.
Examples
Baseline assertion in a smoke test
# in ci/smoke.pyfrom sklearn.metrics import roc_auc_scoremodel, X_val, y_val = train_smoke()auroc = roc_auc_score(y_val, model.predict_proba(X_val)[:, 1])assert auroc >= 0.70, f"smoke auroc {auroc:.3f} below baseline 0.70"A tiny model trains on a subset; if its quick validation metric dips under the baseline the CI fails fast, before a full expensive retrain.
In production
- A team's CI returns in under 15 minutes and every PR shows the smoke AUROC.
- A schema test in CI caught a flipped feature order before the weekly retrain.
- Baseline assertions blocked a dependency upgrade that silently degraded a model.
Best practices
- Keep smoke runs under ~15 minutes so developers actually use CI.
- Pin the fixture dataset and note its version in the evidence bundle.
- Assert on stable evaluation, not training loss curves.
- Make CI failures actionable with the failing metric and expected threshold in the message.
Common mistakes
- Using the full production dataset in CI — slow feedback, expensive pipelines.
- Asserting on training metrics, which always look optimistic.
- Baselines hard-coded in scattered scripts instead of config.
- Ignoring nondeterminism: unseeded smoke runs that flakily pass or fail.
Debugging tips
- 'CI failed on metrics' — rerun with a fixed seed; distinguish flaky noise from a real drop.
- 'Smoke passes but full train breaks' — the full run hits memory/order effects the subset missed.
- 'Schema tests slow' — profile the checks; read the failing row counts rather than the whole frame.
Interview checkpoint
Hands-on lab
- Write a smoke train that runs in minutes
- Add a schema assertion with a pinned fixture
- Assert a stable baseline metric so regression fails CI
Build a Fast ML CI Check
35 min · intermediate
Quick quiz
What is the core value of a smoke train in CI?
Baseline metric assertions belong in CI so silent model regressions fail the build.