Intermediate 14 min read Unsupervised Learning

Unsupervised Anomaly Detection

Find the strange without labels

Anomaly detection flags rows that deviate from the learned 'normal': z-score/statistical rules, IsolationForest's partition-leakage logic, and one-class SVMs — with contamination and threshold choices to manage.

anomaly detectionoutliersIsolationForestz-scoreone-class SVMcontamination

Prerequisites: Unsupervised Learning Overview

Before we start

What you will learn

Choose statistical (z-score/quantile) vs model-based (IsolationForest, one-class SVM) anomaly detectors
Interpret IsolationForest's anomaly score and contour threshold
Tune contamination and evaluate anomalies by business review, not an invented accuracy

Introduction

What is this?

Unsupervised anomaly detection characterizes 'normal' from unlabeled data and flags deviations. Statistical detectors z-score/quantile threshold assuming a distribution; IsolationForest isolates rows by how FEW random splits they survive (anomalies isolate quickly); one-class SVM learns a boundary enclosing normal data and flags anything outside; AutoEncoders reconstruct normal well and anomalies poorly (reconstruction error as score). Anomalies feed alerts, cleanup, and rarely-labelable reviews.

Motivation

Why do you need it?

Real incidents (fraud, sensor failure, intrusion) are rare and poorly labeled — supervised data is scarce by definition.
Detectors adapt as 'normal' shifts, unlike a fixed rule list.
IsolationForest is fast, needs no distance matrix, and scales well.
Anomaly review loops produce the labels that later power supervised models.

Think about it

Mental model

Imagine

Bouncer recognizing regulars and spotting a stranger

Think of

Anomaly detection

The bouncer doesn't need a 'wall of prime suspects' — just a strong sense of who belongs. New faces (anomalies) get a closer look, then a decision.

Mechanics

How it works

Statistical: transform to z-scores or use robust quantiles; flag |z| > threshold or beyond percentiles; robust to distribution assumptions.
IsolationForest: sample sub-forests; features chosen to split recursively; short path to isolation = anomaly. Score near -1 flag.
One-class SVM: find a tight boundary around inlier density with the RBF kernel; outliers fall outside.
AutoEncoder: encode/decode; anomaly = large reconstruction error.
Set contamination for expected anomaly fraction; threshold tuned by business review of flagged rows.

IsolationForest quicksplit logic.

Normalneeds many splits to isolate

Anomalyisolated in few splits

Scorepath length -> anomaly score

Flagscore beyond threshold

Hands-on

Examples

IsolationForest with contamination

import numpy as np
import pandas as pd
from sklearn.ensemble import IsolationForest
from sklearn.model_selection import train_test_split
 
rng = np.random.RandomState(0)
X = rng.normal(0, 1, (800, 4))
X = np.vstack([X, rng.uniform(6, 9, (20, 4))])  # 20 anomalies
 
iso = IsolationForest(contamination=0.02, random_state=0).fit(X)
preds = iso.predict(X)  # 1 normal, -1 anomaly
scores = iso.score_samples(X)
 
# 20 TRUE anomalies are the rows beyond index 800
flags = np.where(preds == -1)[0]
print("flagged count:", len(flags))
print("true anomalies caught:", (flags >= 800).sum(), "of 20")
print("score of mean row vs anomaly row:",
      np.round(scores[:5].mean(), 3), np.round(scores[-5:].mean(), 3))

With contamination=2% the forest flags the far outliers first — their isolation scores plummet compared with the normal cluster's.

Real world

In production

Transaction monitoring: hourly IsolationForest on velocity features flags spikes for human review.
K8s/telemetry: per-metric z-scores detect latency cliffs before SLO breaches.
Manufacturing: sensor reconstruction-error thresholds catch drift before parts fail QC.

Guidelines

Best practices

Fit detectors on a clean-ish historical window; set contamination from historical anomaly rates; validate via human review of flagged rows, precision@review, and recall probes with injected synthetic anomalies; retrain on a cadence matching the drift of 'normal'.

Watch out

Common mistakes

Setting contamination from guesses; thresholding without business review; using Euclidean-based detectors on high-dim data where distances collapse; treating every anomaly as fraud rather than a signal to investigate.

When things break

Debugging tips

Plot the score distribution: a clean bimodal gap means thresholds are easy; a smooth ramp means your data defines no clear normal.
Inject synthetic anomalies to measure recall of the detector.
Drift alert vs anomaly flag confusion — anomaly detectors flag relative rarity, which changes as data shifts.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

IsolationForest scores anomalies how?

contamination parameter roughly means:

Go deeper

Official documentation

IsolationForest (sklearn)

docs

OneClassSVM (sklearn)