Outlier Detection and Treatment
Never delete a story; verify it first
An outlier is either the most interesting data you have or corrupted data wearing disguise. Zone it with IQR/z-scores, sanity-check it against business reality, then carry it, clip it, or flag it.
What you will learn
- Detect outliers via IQR and z-scores
- Verify outliers against business reality
- Choose treatment: keep, clip, flag or research
What is this?
Outlier detection identifies extreme values via IQR fences (below Q1-1.5·IQR, above Q3+1.5·IQR) or z-scores beyond ~3σ. Treatment is the judgment call: verify against facts (is a $40k order possible?), then keep-and-flag (it's the business), winsorize/clip for modeling stability, or investigate a data-error cause. Deleting without verification is the unforgivable move.
Why do you need it?
- One rogue order can shift an A/B lift or a monthly total by double digits.
- Real outliers are often the strategy insight — the largest accounts, the biggest spikes.
- Untreated outliers warp means, regressions and model gradients.
- Deleting without a cause documented is data vandalism.
Mental model
A hospital triage nurse triaging an extreme vital sign
Outlier handling
A blood pressure of 220/110 is not 'ignored so the average looks normal' — the nurse verifies the reading, decides whether it signals real danger or a broken cuff, then treats the case and documents it. Analysts must run the same triage.
How it works
- IQR: compute Q1/Q3; fences = Q1-1.5·IQR and Q3+1.5·IQR; flag rows outside.
- z-score: (x - mean)/std over ~3 flags usually extreme under normality.
- Verify: source lookup, business constraint check ('max order possible?').
- Treat: carry (keep + flag column), clip/winsorize at a chosen bound, or correct a data error identified.
- When the tail is the business (heavy tail), quantiles beat fences — report p99.
Triaging an extreme value.
Examples
Detect, verify-track, clip carefully
import pandas as pds = pd.Series([3, 12, 4, 5, 6, 4, 5, 6, 3, 4, 400])q1, q3 = s.quantile(.25), s.quantile(.75)iqr = q3 - q1fence = q3 + 1.5 * iqrflagged = s[s > fence]print("flagged:", flagged.tolist())print("IQR-based? Only if business allows a $400 order — verify, do not delete.")The IQR fence flags 400; the decision is whether 400 is a real order (carry + flag) or corrupted (investigate) — never silently dropped.
In production
- A fraud detector z-codes transaction amounts and triages the tail for review, not deletion.
- An ML pipeline winsorizes spend at the 99.5th percentile so one whale can't dominate gradients.
- A capacity planner reports p99 latency plus a 'top 5 outliers by day' digest for engineers.
Best practices
- Verify before treating; 'verified as real' is the only license to keep an extreme value.
- Prefer flagging over deleting in shared datasets.
- For modeling, winsorize/clip rather than delete — keeps the sample intact.
- Log every treatment as a decision someone can audit.
Common mistakes
- Silently deleting outliers and shrinking the true distribution.
- Clipping a tail that is the actual business signature.
- Applying z-score fences to heavy-tailed data where ~nothing is 'normal'.
- Treating outliers identically across segments with different natural scales.
Debugging tips
- When deleting changes material results, the 'outliers' were the analysis — flag them.
- z-score naif on multimodal distributions; segment first.
- A '400' that repeats suspiciously → check data-entry or unit mismatch before believing.
- If fences flag 5%+ of the data, your fence definition or distribution assumption is off.
Interview checkpoint
Quick quiz
A value flagged above Q3 + 1.5×IQR is:
Winsorizing replaces extremes with boundary values rather than deleting rows.