Training Pipelines
From notebook epiphany to repeatable operation
A training pipeline is the repeatable, versioned route from data to a trained model: validate input, engineer features, train, evaluate, register. Notebooks are exploration; pipelines are production.
What you will learn
- Describe the stages a training pipeline must run
- Explain why pipelines, not notebooks, are the production unit
- Identify the artifacts a pipeline must produce
What is this?
A training pipeline is a scheduled, versioned sequence of steps — validate data, build/verify features, train, evaluate, register a candidate — that reliably converts a data version into a model version. It packages the experiment's magic into a deterministic operation anyone can run and audit.
Why do you need it?
- Notebook state is not reproducible; a pipeline has inputs and outputs you can pin and replay.
- Retraining is only trustworthy if it runs the same steps every time.
- A pipeline integrates with data quality gates, the registry and triggering automation.
- Failures are step-localized instead of 'the notebook stopped at some cell'.
Mental model
A recipe's mise en place checklist
Training pipeline stages
Pro chefs prep validated ingredients, cook in fixed order and plate from a defined recipe. A pipeline is the kitchen discipline: verify produce (data), prep (features), cook (train), taste (evaluate), plate (register) — in order, repeatably.
How it works
- Trigger: schedule, new data version, drift alert, or manual run.
- Validate inputs: schema, quality gates, expected row counts.
- Feature step: build/verify the feature matrix against the data version.
- Training step: fit the model with pinned config and seeds on the compute allocation.
- Evaluation: measure against held-out sets and the champion baseline.
- Register: write the winning artifact + metadata to the model registry.
Stages of a repeatable training pipeline.
In production
- A nightly pipeline retrains churn using the newest labeled day; registrations feed the promotion gates.
- A drift alert triggers a training run on demand through the same pipeline.
- A data-quality failure halts the pipeline before any training GPU is wasted.
Best practices
- Make each step idempotent and step-localized (inputs → outputs explicitly).
- Pin data version, config and seeds at the pipeline level.
- Let the pipeline register candidates; humans/champions handle promotion.
- Instrument every step with metrics and artifact hashes.
Common mistakes
- Repackaging a notebook into a script and calling it a pipeline.
- Step order determined by race conditions instead of a DAG.
- Silently 'continue on missing data' — training on whatever fell through.
- No registry output, so the pipeline's result vanishes.
Debugging tips
- Pipeline failed at a step? Repro that step in isolation with its pinned inputs.
- Same inputs, different output? Diff seeds, ordering and env versions across runs.
- Slow step? Profile that step; the pipeline exposes it now that it's a step.
Interview checkpoint
Quick quiz
Which stage in a training pipeline protects against training on corrupted input?
A training pipeline should register its candidate model with evidence in the registry.