Advanced 14 min read Unsupervised Learning

Autoencoders for Compression and Outliers

Learn the data's own codebook

An autoencoder learns to compress and reconstruct its input; reconstruction error doubles as both a learned dimensionality reduction and an anomaly score that adapts to the data.

autoencoderencoder-decoderreconstruction errorbottleneckrepresentation learning

Prerequisites: Anomaly Detection

Before we start

What you will learn

Describe the encoder-bottleneck-decoder architecture and its training loss
Use reconstruction error as a learned anomaly score
Choose compression degree and recognize over- vs under-complete tradeoffs

Introduction

What is this?

An autoencoder is a neural network trained to reproduce its input: an encoder squeezes X into a small latent bottleneck z, a decoder expands z back. Minimizing reconstruction error (MSE) forces z to carry the important, compressible summary while noise is discarded. Applications: dimensionality reduction that beats linear PCA on manifolds, denoising (trained on corrupted inputs), and anomaly detection via reconstruction error.

Motivation

Why do you need it?

Learns non-linear, data-adapted features — PCA is its linear special case.
Reconstruction error is interpretable and label-free: normal rows rebuild well, novel/aberrant rows don't.
Denoising-variant autoencoders clean sensor and imagery noise before downstream ML.
Latent z is a ready-made embedding for retrieval and cold-start features.

Think about it

Mental model

Imagine

Summarizing a book in a hundred words, then retelling it faithfully

Think of

The bottleneck autoencoder

The hundred-word summary (z) forces the essential plot to be captured; if retelling drifts, the important parts were missed. Rows a reteller reconstructs badly are the ones the summary model never understood — anomalies.

Mechanics

How it works

Architecture: dense encoder layers -> small latent (bottleneck) -> dense decoder layers -> input dims.
Train with MSE between input and reconstruction; optimizer minimizes both halves together.
Under-complete z (smaller than input) forces compression; over-complete with regularization can copy identity.
Denoising: corrupt inputs during training (noise), target clean; the model learns robust features.
Anomaly score: for a new row, reconstruction error; high error = poor fit to learned normal.

Autoencoder flow.

EncoderX -> z bottleneck

Bottleneckcompressed code

Decoderz -> X'

LossMSE(x, x') drives representation

Hands-on

Examples

Reconstruction error separates injected outliers

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
 
rng = np.random.RandomState(0)
X = rng.normal(0, 1, (1000, 8))
X[:20] = rng.uniform(5, 8, (20, 8))  # 20 anomalies mixed in
Xs = StandardScaler().fit_transform(X - X.mean(0))
Xs = np.nan_to_num(Xs)
 
import torch
import torch.nn as nn
import torch.optim as optim
 
Xin = torch.tensor(Xs, dtype=torch.float32)
model = nn.Sequential(
    nn.Linear(8, 6), nn.ReLU(), nn.Linear(6, 3), nn.ReLU(),
    nn.Linear(3, 6), nn.ReLU(), nn.Linear(6, 8),
)
opt = optim.Adam(model.parameters(), lr=0.01)
for _ in range(300):
    opt.zero_grad()
    loss = ((model(Xin) - Xin) ** 2).mean()
    loss.backward()
    opt.step()
 
err = ((model(Xin) - Xin) ** 2).mean(1)
r = np.argsort(-err.detach().numpy())
print("true anomalies are rows 0-19;"
      " top-20 by reconstruction error contain:", sorted(set(r[:20]) & set(range(20))))

After training on the whole (mostly normal) set, injected outliers show the largest reconstruction errors — the compact code captures normal variation and fails to rebuild odd rows.

Real world

In production

Telemetry: sensor autoencoder flags rows whose reconstruction jumps — earlier and smarter than static z-score rules.
Retrieval: the latent vector z is the embedding for semantic search over tabular/image content.
Denoising: image and audio denoisers trained on corrupted inputs before fed to classifiers.

Guidelines

Best practices

Standardize inputs; use under-complete bottlenecks for compression; monitor validation reconstruction MSE for overfit; for anomaly thresholds, fit a quantile on training-row errors; compare gains against PCA baselines before adopting.

Watch out

Common mistakes

Over-complete + no regularization lets the net copy inputs (identity — useless compression); reading MSE headlessly without inspecting WHAT reconstitutes badly; assuming 2-hidden-layer is 'enough depth' for rich data; not splitting normal from anomaly windows when training.

When things break

Debugging tips

Plot reconstructed vs original for a handful of rows — blurs tell you the bottleneck is too tight.
Watch train/valid MSE gap: big gap means the codebook memorized training specifics.
Compare recomputation across random seeds — highly variable latent codes signal under-training or too-large lr.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

The bottleneck's role is to:

An autoencoder trained on normal telemetry flags anomalies via:

Go deeper

Official documentation

Autoencoder tutorial (Keras guide)

docs

Autoencoder | Wikipedia