Anomaly Detection for Data
Catching the drift that thresholds can't see
Static thresholds miss silent drift. Anomaly detection builds baselines from history and flags distributions that move — volume, metric and value-profile drift — before they become worse than a hard failure.
What you will learn
- Model baselines (mean/std, seasonality) over historical data per table
- Distinguish hard-rule failures from statistical anomalies and tune sensitivity
- Design a drift workflow: detect, validate, escalate or auto-quarantine
What is this?
Anomaly detection in data quality learns the normal range of a metric over time — row counts, sums, null rates, delays — including weekly seasonality and trend — then scores each new observation against that expectation. When a metric moves far outside its modeled range (e.g. row count down to 10% of seasonal average), it's flagged as an anomaly even though no fixed threshold was crossed. It complements hard assertions: assertions catch binary violations; anomaly detection catches subtle drift that slowly poisons models and reports.
Why do you need it?
- Static thresholds go stale: businesses grow, seasons change, and '5% drop' becomes noise or disaster depending on when.
- Many serious defects are gradual — fewer events last week, skewing A/B results — invisible to any one hard check.
- Automated baseline modeling scales across thousands of tables where hand-tuned thresholds don't.
- Anomalies are early warnings: by the time a hard threshold trips, the damage is already in dashboards.
Mental model
A river gauge with expected seasonal highs and lows
Anomaly detection on metrics
Flood predictions don't use one fixed water level year-round; they model the seasonal norm and alert when the level strays far from what's expected for this week. Data quality metrics deserve the same seasonal awareness.
How it works
- Collect metric history per dataset/column: volume, null rate, sum, mean, distinct count, delay, freshness.
- Model a baseline: rolling mean/std, exponential smoothing, or a seasonality-aware model (weekly/daily).
- Score each new point: z-score or deviation ratio against expectation with a tolerance band.
- Tune sensitivity per dataset: alert on flag only above a probability/severity, or auto-quarantine after N consecutive flags.
- Emit anomaly events (metric, expected, observed, window) to the alerting channel and catalog.
- Require human feedback (true/false positive) to continuously tighten each model.
Baseline band and the anomaly that pokes through it.
In production
- A subscription business watches daily active events; a 30% drop flagged as an anomaly turns out to be a tracking SDK that stopped firing — caught in hours, not a week.
- A logistics warehouse flags spent-hours anomalies by day-of-week, so a holiday dip isn't paged as a failure.
- A loyalty program auto-quarantines a nightly aggregate when it's 5 sigmas from expected for two consecutive runs, preventing a wrong payout file.
Best practices
- Model seasonality (day-of-week, month) before applying threshold deviations.
- Start with volume and freshness anomalies — they catch the most impactful silent failures.
- Feed human verdicts back into the model (sensitivity per dataset) to kill alert fatigue.
- Pair anomaly detection with hard assertions; anomalies without a hard rule can be softer noise.
Common mistakes
- Applying weekly-seasonal expectations to a weekend-flat stream and paging every friday dip.
- Setting the band so tight that normal variance becomes alert spam (2-sigma on noisy data).
- Using only mean/std on skewed metrics — these mask long-tail misshapenness.
- Forgetting trend: a growing dataset outruns last-year's baseline inside a quarter.
Debugging tips
- When an anomaly fires, plot expected vs observed for the past 28 days first — context cures false scares.
- Cross-reference the anomaly window with deploys and source changes on the same day.
- Bin by dimension before judging: a global 2% drop might be a 40% drop in one country.
- Record the model's inputs at flag time; a 'data window shifted' root cause often surfaces half a second later.
Interview checkpoint
Hands-on lab
- Build a seasonal baseline from 60 days of daily revenue
- Score new days with a z-score-style deviation and a tolerance band
- Catch an injected 8% drift that a fixed threshold would miss
Detect a Sales Metric Anomaly with a Baseline
35 min · advanced
Quick quiz
Which signal is an anomaly detector best at catching?
Anomaly detection replaces hard data assertions.