Data & Concept Drift

• Distinguish data drift, covariate shift and concept drift
• Choose detection methods (PSI, KS, chi-square) and windows
• Turn drift signals into concrete operational actions

• Drift is the leading indicator: quality drops later, users feel it first.
• Confusing the two types sends you fixing the wrong thing (retraining a model when the target changed policy-wise).
• Statistical 'drift' on shift-vs-noise is how you avoid retraining storms on benign seasons.
• Knowing which features drifted tells you whether to retrain, add data, or redesign the feature.

If you move the forecast inland (data drift), the same typical-temp model now predicts badly — P(weather) changed. If the region's seasons flip (concept drift), even identical summer conditions now mean different storms — P(outcome|conditions) changed. Both require a new forecast, but the fix is different: more local data versus learning a new relationship.

• Window the inputs: training/reference distribution vs rolling production window (day, week).
• Measure per feature: PSI (population stability index), KS test, chi-square for categoricals, or model-based detectors.
• Detect concept drift: compare expected vs observed outcomes on a delayed-label cohort; use adaptive window algorithms (ADWIN) for streaming data.
• Filter: separate real structural drift from seasonal/benign shifts using business context and calendar windows.
• Act: retrain trigger, data-collection gap fill, feature-space expansion, or a decision-threshold recalibration.

• A loan-approval model tracks PSI per feature weekly; when 'income' shifts > 0.25 PSI, a data-collection task starts before retraining (the shift is often a real economic change, not an error).
• A churn model uses a KS test on a 7-day window; each detector is paired with a runbook — 'field maintenance', 'promo war', 'retrain'.
• An anomaly system uses ADWIN to adapt windows online so concept drift updates its baseline instead of drowning in false peaks.

• Detect per feature AND on the joint/prediction — one changed feature can be noise; the prediction distribution rarely lies.
• Pair every detector with a decision rule and owner; an alert without an action is decoration.
• Reconcile drift with true labels later: drift + confirmed concept change validates retraining triggers.
• Benchmark drift indicators against known historical incidents so thresholds encode real events.
• Use calendar-and-quantile windows to make seasonality expected, not an alert.

• Retraining on every drift alert without asking whether the shift is benign, real, or a pipeline bug.
• Treating data drift and concept drift identically when their fixes differ (more data vs new model vs recalibrate).
• Alerting on z-score per feature with no multiple-comparison guard — with 50 features, noise trips alerts constantly.
• Measuring drift on processed features the model sees, missing breakage upstream in raw data.

• Look at drift per feature vs drift in predictions: if features shifted but predictions didn't, the model is robust or ignoring them — check both.
• Confirm the drift is in raw data, not a pipeline bug (e.g., a field mapping change) before opening a retrain ticket.
• On concept drift, examine label-policy/game changes (fraudsters adapt, incentives change) before blaming the model.
• Backdate a detector's output vs known incidents to validate its sensitivity before relying on it.