Drift on Numerical Features
When numbers move — means, scales and tails
Numeric features drift through mean shifts, scale changes, or tail movements. PSI, KS tests and distribution distances catch them — but binning choices and reference windows determine whether the number you get reflects reality or noise.
What you will learn
- Describe how numeric drift manifests (location, scale, shape)
- Apply PSI and KS to numeric features
- Design bins and reference windows for stable detection
What is this?
Numeric drift is a change in a continuous feature's distribution — its mean (location), spread (scale), or tail shape — versus training. Detectors like PSI over bins and the KS test over cumulative curves flag it, but the choice of binning and reference window decides whether the flag is true.
Why do you need it?
- A mean-shifted feature (income up 20%) silently changes every prediction that uses it.
- Scale drift loosens or tightens confidence in ways hard to eyeball.
- KS catches shape/tail changes PSI's coarse buckets can miss.
- Per-feature numeric drift monitors guide targeted retraining.
Mental model
A fishing net's mesh size vs a watermark line
PSI vs KS on numeric drift
Buckets are mesh — coarse, cheap, catches gross shifts. KS scans the water-line curve — sensitive to a swelling tail that shifts total pool (distribution). Choose the mesh by what you can afford to check.
How it works
- Bucket the numeric feature into reference bins (deciles/quartiles or chosen cuts).
- PSI compares reference bin proportions to live proportions.
- KS computes the max gap between cumulative distributions.
- Mean/scale stats track alongside as interpretable supplement.
- Thresholds calibrated per feature trigger drift flags.
Numeric drift through the eyes of PSI and KS.
In production
- Income feature PSI crossing 0.2 triggers a check that precedes a fraud decay.
- KS flags a fat contain_tail in request volumes that mean-drift missed.
- A scale-shift on a score feature prompts clipping before it warps else-skipped logic.
Best practices
- Use stable reference buckets and freeze them; live data re-buckets identically.
- Pair PSI with KS where tail sensitivity matters.
- Include mean and std stats for human-readable context on alerts.
- Re-baseline deliberately when the model retrains, documenting the reason.
Common mistakes
- Dynamic bin edges per window — comparing apples to oranges.
- Borrowing 0.25 as a universal PSI threshold.
- Small live windows spiking KS on noise.
- Only mean-tracking, missing tail-driven drift.
Debugging tips
- PSI says drift, KS quiet, mean stable — check bin-boundary artifacts; the buckets may just be coarse.
- KS flutter — shrink the live window or collect larger samples before reacting.
- Both say drift on one feature of many — trace its upstream; a provider schema change is classic.
Interview checkpoint
Hands-on lab
- Detect mean/scale shifts with simple stats
- Apply a KS-style max-gap comparison
- Classify drift with baseline-calibrated thresholds
Detect Numeric Drift
35 min · intermediate
Quick quiz
For a numeric feature, drift can show up as…
Freezing reference bins makes drift comparisons consistent over time.