Data Drift
When the inputs change under the model's feet
Data drift is when the inputs your model sees in production start looking different from training: new ranges, shifted distributions, or new categories. It's the first warning of decay — the model still runs, but on data it never learned from.
What you will learn
- Define data drift in terms of training vs production distributions
- Distinguish data drift from concept drift
- Recognize drift's ripple effects on predictions
What is this?
Data drift is a change in the distribution of model inputs between training and production — new ranges, shifted means, novel categories, new missingness — that makes a static model solve for a world that has quietly changed. It's the earliest, most common signal of production decay.
Why do you need it?
- Models are frozen at train time; the world they predict keeps moving.
- Input drift propagates: the outputs you rely on shift even when the model code is fine.
- Catching drift early means retraining on the right window, not reacting to damage.
- It's label-free — you can detect it without waiting for ground truth.
Mental model
A chef trained on coastal seafood suddenly in a landlocked market
Data drift
The chef (model) still cooks perfectly — but the ingredients (inputs) arriving are different fish, smaller quantities, new seasons. Same skills, wrong menu, guaranteed wrong results until the menu is relearned for the new market.
How it works
- Training sets a reference distribution per feature.
- Production windows are sampled continuously or on schedule.
- Statistical distance tests compare production to reference.
- Breaches over threshold raise drift alerts.
- Drift flags feed retraining decisions and windows.
Detecting drift between reference and live windows.
In production
- A churn model alerts when the age distribution of signups shifts 2σ from training.
- A fraud model flags a novel transaction category the classifier never saw.
- A recommender detects seasonality drift between its holiday and weekday training windows.
Best practices
- Detect drift label-free so it fires before damage
- Monitor per feature and overall — one feature can carry a whole model.
- Set thresholds from baseline variability, not constants.
- Pair drift alerts with a retraining path ready to act.
Common mistakes
- Treating drift as 'maybe suspicious' with no action policy.
- Only monitoring outputs; inputs move first.
- Alerting on noise by using too-short windows.
- Retraining into the drifted distribution without checking it's real.
Debugging tips
- Drift alert but business fine — verify the drift is real and not a logging/pipeline change.
- Single feature drifting — trace its upstream sources; a schema or provider change is common.
- Drift everywhere — suspect a data pipeline change, not the world.
Interview checkpoint
Quick quiz
Data drift is fundamentally a change in…
Data drift can be detected before ground-truth labels arrive.