Statistical Tests and Hypothesis
The statistics under the drift alarm
Drift detection is hypothesis testing in disguise: null = 'no shift', a p-value says whether the evidence clears the bar. The discipline — sample size, multiple comparisons, effect vs significance — is what keeps drift alerts statistically honest.
What you will learn
- Frame drift detection as a hypothesis test
- Interpret p-values without idolizing them
- Handle multiple comparisons and small samples
What is this?
Statistical tests underpin drift detection: the null hypothesis says live and reference come from one distribution; tests (KS, chi-square, AD) measure evidence against it. Doing this honestly means sizing samples, correcting for many features tested at once, and reading effect size, not just the p-value.
Why do you need it?
- Every drift alert is a statistical claim — it should stand up as one.
- Testing 50 features and alerting on any p<0.05 alarms on randomness by design.
- Statistical significance ≠ practical importance; effect size matters.
- Small samples make tests noisy; honest savings prevent false fire drills.
Mental model
A courtroom's beyond-reasonable-doubt standard
Significance testing
The null is 'innocent until proven'. Evidence (p-value) must clear a high bar before conviction (alert), and trying 50 defendants in one trial means the standard has to tighten (multiple corrections). Verdict = significance; sentence = effect size.
How it works
- State the null: no drift. Choose a test matching the feature type and assumptions.
- Collect a reference and a live sample sized for the test and expected effect.
- Run the test to a p-value; compare against a pre-set alpha.
- Correct for multiple comparisons across features (Bonferroni/FDR).
- Report effect size alongside significance; thresholds gate the alert.
Hypothesis testing loop for drift.
In production
- A drift suite runs KS on 30 features with FDR correction so true alerts survive noise.
- A small-but-significant dip is read with its effect size before a retrain gets triggered.
- Sample-size planning stops underpowered windows from crying wolf.
Best practices
- Set alpha before looking; adjust for the number of tests.
- Require both significance and a minimum effect size for alerting.
- Plan sample sizes so the test can actually detect the drift you care about.
- Log test parameters (test, alpha, correction, effect) with every alert.
Common mistakes
- Alerting on any p<0.05 across 100 features — guaranteed noise fires.
- Ignoring effect size: 'significant' with a trivially small shift.
- Tiny windows producing unstable p-values and flapping alerts.
- Re-baselining after every alert so the null keeps moving.
Debugging tips
- Lot of borderline p-values — check you applied multiple-comparison correction.
- Alert hovering at p=0.049 — that's noise range; require effect size too.
- Underpowered test — increase window size or lower the effect you promise to detect.
Interview checkpoint
Quick quiz
When monitoring many features, you must…
A small-but-significant drift is always worth retraining for.