ML Lifecycle
Intermediate 14 min read ML Lifecycle

From Data to Model

The workflow where silent mistakes are born

The supervised workflow — explore, prepare, train-validate-test, evaluate — is where most mistakes hide. Divides invisible in the notebook (leakage, shuffle chaos) decide whether the model generalizes or lies.

train test splitdata leakagevalidationsupervised learning workflowfeature engineeringevaluation
Prerequisites: The ML Lifecycle Scoping ML Projects
Before we start

What you will learn

  • Order the supervised workflow and the purpose of each split
  • Explain temporal versus random splitting for time-series data
  • Catch the most common eval mistakes before they reach production
Introduction

What is this?

The data-to-model workflow is the path from raw data to a trustworthy evaluation: exploration, cleaning, feature engineering, an honest train/validation/test split, training, and evaluation on a held-out set. Its discipline determines generalization — a model that 'performs great offline' and fails live is almost always a split or leakage failure.

Motivation

Why do you need it?

  • Models memorize; honest evaluation is the only guard that what you built is prediction, not pattern matching.
  • Random splits lie on time-ordered data — using the future to predict the past inflates scores magically.
  • Leakage (target information in features) is the #1 cause of 'impossibly good' offline numbers.
  • The split defines the whole downstream lifecycle: every retrain and every registry entry inherits its trustworthiness.
Think about it

Mental model

Imagine

Mock exams before the real test

Think of

The train/validation/test workflow

You practice on past papers (train), tune strategy on a real mock (validation), and only then sit the actual exam (test). Using the real exam answers during practice is 'leakage' — and it's exactly what makes ML scores lie.

Mechanics

How it works

  • Exploration checks shapes, missingness, distributions and target balance before any modeling.
  • Cleaning fixes types, duplicates and impossible values; every change is documented as code.
  • Feature engineering adds transformed inputs; domain knowledge lives here.
  • Splitting holds out evaluation data: a validation set steers training, a test set is touched only once.
  • Training fits the model; evaluation reports performance on the held-out sets and against the baseline.
  • Every artifact (code, features, split, metrics) is recorded so the result is reproducible and auditable.

The supervised workflow with honest evaluation.

Raw data
Explore & clean
Feature engineering
Split
Train
Evaluate on test
Hands-on

Examples

Leakage in a 30-second glance

Leakage in a 30-second glance
# BAD: target appears inside features
df['approved'] = df['credit_score'] > 700   # leakage!
features = df.drop('approved', axis=1)
X_train, X_test, y_train, y_test = train_test_split(features, target, test_size=0.2)
 
# GOOD: split by time for time-ordered data
cutoff = '2026-06-01'
train = df[df.tx_date < cutoff]
test  = df[df.tx_date >= cutoff]
# fit scaler and encoder on train only, transform both consistently

Leakage inflates offline metrics because the model 'sees' the target. Temporal splits prevent the future leaking into the past. Fit any scaler/encoder on training data only and reuse that fit for validation and test.

Real world

In production

  • A fraud model splits by transaction time, not user, because users can recur across train and test.
  • A churn model uses group-aware splits so repeated customers aren't in both train and test — avoiding optimistic correlation.
  • Every promotion to a registry carries its split recipe and held-out metrics, so the registry trusts the eval.
Guidelines

Best practices

  • Split by time (or a group key) whenever rows are ordered or grouped; use random split only for stationary i.i.d. data.
  • Fit preprocessing on train only; copy that fitted transform to validation/test to avoid skew.
  • Record the random seed, split recipe and the exact data version used.
  • Check target leakage explicitly: rerun feature importance and look for 'too perfect' features.
Watch out

Common mistakes

  • Using a random split on time-series data and 'beating the market' in offline eval.
  • Scaling validation/test data with their own fitted scaler.
  • Tuning hyperparameters on the test set until it's just a second validation set.
  • Engineering features from the whole dataset (including the future) before splitting.
Root cause analysis

Common issues

Error
Offline AUC 0.99, production performance collapses
Root cause

Leakage or group-overlap in the split inflated the offline number

Fix

Inspect top features for target-derived values; switch to temporal/grouped splits.

Error
Validation improves but test is flat
Root cause

Test set re-used for tuning, so it no longer measures generalization

Fix

Freeze the test set; tune on validation only; re-split only when genuinely needed.

When things break

Debugging tips

  • Suspiciously high score? Hunt for leakage: sort features by importance and inspect the top ones.
  • Reproduce by re-running the split with a fixed seed and diffing the eval numbers.
  • Check whether entities (users, sessions) leak across splits — the common group-wise error.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which split is appropriate for a churn model over 2 years of customer data?

2

A feature derived from the target is a serious but common source of leakage.

Go deeper

Official documentation

scikit-learn — train_test_split
docs
Google — data splits & leakage
docs