The ML Lifecycle

• Model the full ML lifecycle as a loop with explicit stage boundaries
• Identify the artifacts and tests each stage must produce
• Locate where most production ML failures actually originate

• Thinking 'train → ship → done' hides where systems really fail: data quality, monitoring and decay.
• Missing stage checkpoints destroy reproducibility — you cannot retrain what you didn't version.
• The feedback arrow (monitoring → retraining) is what keeps a model honest as the world changes.
• Framing the right problem first avoids building an excellent model for the wrong question.

Planting (training) is one week. Soil testing (data verification), irrigation and pest watches (monitoring), and deciding next season's crop (retraining) run all year. A farmer who only thinks about planting week fails; so does an ML team that only thinks about training week.

• Frame: define the business goal, success metric, and what a prediction actually changes.
• Collect & verify data: gather raw sources, check schema, distribution, label quality; freeze dataset versions.
• Train & evaluate: build features, train candidates, compare on holdout and business metrics, register the winner.
• Deploy: promote through registry gates — shadow, canary, then production.
• Monitor: track drift, latency, data quality and a label-proxy for model quality.
• Retrain: triggered by monitors or a freshness schedule; the new candidate re-enters the loop.

• A fraud team keeps a weekly retraining cadence driven by label arrival (disputes arrive days after transactions), with drift alerts accelerating the loop.
• A search ranking system monitors relevance proxies hourly and backtests a candidate weekly before promotion.
• A healthcare risk model runs the whole loop under audit control — every stage leaves a trace for regulators.

• Version the dataset and features at the data stage; every training run names its exact inputs.
• Exit each stage only with tests: schema checks in data, metric thresholds in train, health checks in deploy.
• Define monitoring in the deploy stage, not after an incident.
• Make retraining trigger-driven and schedule-based hybrid — no manual 'it feels stale' refreshes.
• Document the decision at every gate: who, what evidence, when.

• Spending 90% of effort on training and treating monitoring as an afterthought.
• Retraining without backtesting against past data — the new model may repeat the old failure.
• Monitoring model quality with delayed labels and never reconciling predictions against outcomes.
• Framing changes mid-project so the model optimizes a goal that stopped mattering.

• Check stage-boundary artifacts when something fails: did the data pass its tests, did the model register?
• A 'sudden' production drop is usually gradual drift the monitors missed — backdate the monitoring.
• Diff the training distribution vs the current live inputs directly; the gap explains most decay.
• Re-run the failed stage's tests, not the whole loop, to isolate where a break is.