Drift on Categorical Features
Categories move in mixes and surprises
Categorical drift shows up as new categories, shifted proportions or rare values blooming. Chi-square tests compare observed-vs-expected counts; JS divergence measures distribution distance; the novel-category rate is a standalone red flag.
What you will learn
- Detect proportion-shift, rare-value bloom and novelty in categories
- Apply chi-square and JS divergence correctly
- Handle unseen categories at serving time
What is this?
Categorical drift is when a discrete feature's value mix changes: known categories shift in proportion, rare values bloom, or entirely new categories appear that training never encoded. Chi-square and JS divergence quantify it, and new-category rate is a separate, cheap early-warning.
Why do you need it?
- A category that dominated training can vanish, gutting the model's decision surface.
- Novel categories break one-hot/embedding encodings at serving time.
- Rare-value bloom is often the earliest sign of data-quality decay.
- Category mixes are cheap to monitor and highly interpretable.
Mental model
A vending machine restocked with the wrong mix
Categorical drift
The machine (model) only knows the categories it was stocked with. When the mix changes — gamers buy 80% energy drinks, or a brand it never carried shows up — the machine either mis-predicts demand or jams on the unknown brand.
How it works
- Reference stores the training proportion per category (with smoothing).
- Chi-square compares observed live counts to expected reference proportions.
- JS divergence measures the distribution distance between mixes.
- Separately track the rate of categories never seen in training.
- Thresholds per feature flag drift and surface the top-shifted categories.
Categorical drift pipeline.
In production
- A lending model pages when the region mix shifts 30% vs training.
- A support router detects new intent categories nobody encoded.
- A recommender's rare-item bloom triggers data-quality review before predictions skew.
Best practices
- Smooth categories (add-1/Laplace) so rare ones behave in tests.
- Freeze the reference mix; re-baseline only by deliberate retraining.
- Always surface top-shifted categories with the alert — a number without causes is useless.
- Plan the serving behavior for unseen categories before they arrive.
Common mistakes
- Chi-square on categories with near-zero expected counts — the test goes unstable.
- Ignoring novelty until it breaks the encoder at request time.
- Thresholds that only watch total distance, hiding a single category collapse.
- Re-baselining the mix silently every window — 'drift' becomes impossible.
Debugging tips
- Chi-square spiking — list per-category residuals; the pattern (one giant mover) explains it.
- Novel category hit at serving — checks for a fallback 'unknown' path in the encoder.
- Rare-value bloom — check upstream data cleaning; often a parsing or ingestion bug.
Interview checkpoint
Hands-on lab
- Implement a chi-square-style comparison
- Track novelty rate of unseen categories
- Surface top-shifted categories
Detect Categorical Mix Drift
35 min · intermediate
Quick quiz
The earliest, cheapest sign of categorical trouble is often…
Chi-square needs reference proportions with adequate expected counts to be stable.