Drift Detection Methods
Measuring how far the new data is from training
Detectors compare live and reference distributions: PSI and KS for numbers, chi-square/JS for categories, and model-based tricks for high-dimensional data. No detector is universally 'right' — each trades sensitivity, compute and interpretability.
What you will learn
- Match detectors to feature type
- Interpret PSI and KS output and thresholds
- Trade off sensitivity vs noise in detector choice
What is this?
Drift detection methods quantify distributional distance between a reference window and live data: PSI and KS tests for continuous features, chi-square and JS divergence for categorical, and model-based detectors (classifier-as-detector) for high-dimensional or unstructured inputs.
Why do you need it?
- You need a number, not a feeling, to gate retraining decisions.
- Different feature types violate different statistical assumptions.
- Detectors vary in sensitivity — the right one catches real drift without alerting on noise.
- Monotoring across detectors gives per-feature explainability.
Mental model
Comparing a crowd's demographics to the town's census
Distribution-distance detectors
Age histograms shifted? KS/PSI tells you how far. Hair-colour categories changed? Chi-square tells you whether the mix is plausible. The census row is the reference; tonight's crowd is the live window.
How it works
- Bucketed continuous features: PSI measures proportional-shift between reference and live bins.
- Cumulative distribution curves: KS detects location/shape change.
- Categorical: chi-square tests observed-vs-expected counts; JS divergence measures distance.
- Model-based: a classifier trained to tell reference vs live apart scores drift when it wins.
- Results cross thresholds → per-feature drift flags.
Match detector to feature type.
In production
- A credit model runs PSI per numeric feature weekly; income PSI > 0.2 pages the owner.
- A fraud model uses chi-square on category mixes to catch novel-token explosions.
- An embedding model uses a classifier-as-detector because per-feature tests don't scale.
Best practices
- Pick the detector that matches the feature's distribution and your sensitivity need.
- Calibrate thresholds on baseline windows, not folklore constants.
- Age buckets deliberately; naive bins flatter or exaggerate shift.
- Combine per-feature tests with an overall drift summary.
Common mistakes
- Using PSI on categorical or KS on tiny samples and misreading results.
- Thresholds borrowed from another model wholesale.
- Zero-bin handling ignored — empty buckets blow up ratios.
- Detecting drift everywhere but nowhere to act.
Debugging tips
- PSI spikes on bin-boundary movement — the drift may be a bucketing artifact.
- Small reference sample → repeated false alarms; expand the reference window.
- Detector disagreement — compute both distances; one method often clarifies the other.
Interview checkpoint
Hands-on lab
- Implement PSI for a bucketed feature
- Run a KS-style comparison
- Set a threshold from a true baseline
Compute Drift Distances
35 min · intermediate
Quick quiz
PSI is best suited to…
Drift thresholds should be calibrated per feature and baseline.