CI/CD for ML

• Design CI stages for ML: code checks, data checks, train+evaluate
• Explain CT (continuous training) and its trigger rules
• Gate promotions on automated evidence, not vibes

• Models fail on data, not just code; CI must test the data and the train-evaluate path, not only the syntax.
• Automated gates replace 'trust me, the notebook worked' with reproducible evidence.
• Faster, safer cadence: small changes shipped often instead of one giant risky retrain.
• CT keeps a deployed model honest when the distribution moves slowly.

Every car passes inspection before it rolls off (CI). New parts are tested in prototype (staging + shadow). If a batch of steel arrives different (data drift), the line doesn't keep stamping old tolerances — it recalibrates (CT) and every new car is re-inspected against updated specs.

• CI stage 1 (code): lint, type-check, unit tests on feature/model code.
• CI stage 2 (data): schema validation, range/stat checks, leakage and label-quality tests on the frozen dataset version.
• CI stage 3 (train & eval): smoke training run; compare holdout metrics to a registered baseline; gate passes/fails.
• CD: on green, register the artifact and promote through registry stages (staging → production) with approval.
• CT: a scheduled or drift-triggered job re-runs the train-eval pipeline and opens a candidate when thresholds improve.

• A fraud team's PR runs a 5-minute evaluation on a fixed sample; PRs that beat the registered decision-point AUC by 0.01 are auto-promotable.
• A recommender ships every Tuesday using the same pipeline that ran last Tuesday — identical stages, new data, comparable gates.
• A churn model retrains quarterly and on drift triggers, always backtesting before registry promotion.

• Fix the evaluation protocol (dataset, split, metrics) so gates compare apples to apples across candidates.
• Run data checks BEFORE the (expensive) training stage — fail cheaply.
• Gate promotion also on non-metric signals: schema compatibility, latency budget, calibration.
• Keep the CI dataset representative of production; a stale 'golden' set decays in value.
• Make retraining (CT) reuse the same pipeline as interactive training — one code path.

• CI that only lints code — the model-quality gate is where actual value is checked.
• Comparing candidates against a moving baseline without a frozen reference run.
• Re-training on every commit (expensive, noisy) instead of on meaningful change or drift.
• Gate thresholds tuned to pass today instead of protecting against regressions tomorrow.

• When CI fails training but passes locally, check commit-vs-data pairing and dependency changes first.
• A gate that suddenly flips red often means the CI dataset or label definition changed silently.
• Keep baseline reference runs as artifacts; use them to diff candidate metrics precisely.
• Log the exact commit + dataset + seed per CI run; irreproducible CI ever is worse than no CI.