Intermediate 14 min read Unsupervised Learning

PCA and Dimensionality Reduction

Compress the feature space, keep the signal

PCA finds orthogonal directions of maximum variance, projecting high-dimensional data onto few components — denoising, compressing, and enabling visualization, at the cost of interpretability.

PCAprincipal componentsvarianceeigenvaluesdimensionality reductionscree plotSVD

Prerequisites: Unsupervised Learning Overview

Before we start

What you will learn

Explain what principal components are and why variance ordering matters
Read a scree plot and pick component count from explained variance
Know when PCA helps (visualization, denoising, decorrelation) and when it hurts (interpretability)

Introduction

What is this?

Principal Component Analysis computes the eigenvectors of the covariance (or correlation) matrix, sorted by eigenvalue = variance explained. They're the orthogonal directions of maximum variance, and projecting data onto the top k principal components gives the best k-dimensional linear approximation (for squared loss). Standard practice: scale features first, so variance isn't dominated by unit choice.

Motivation

Why do you need it?

Visualization: 2-3 PCs let you SEE high-dimensional structure.
Denoising: components with tiny variance are often noise; dropping them cleans the signal.
Decorrelation: uncorrelated PCs stabilize linear models and remove multicollinearity.
Compression: fewer components mean faster training and smaller clones.

Think about it

Mental model

Imagine

Photographing a mountain range from exactly the most informative angles

Think of

Projecting data onto principal axes

From many random angles you see clutter; from the principal viewpoints the structure (longest dimension, then the next) is obvious. PCA finds those angles and keeps the top few viewpoints.

Mechanics

How it works

Center and scale features (standardize or similar) so variance is comparable.
Compute covariance matrix; find its eigenvectors/eigenvalues (or SVD directly).
Sort eigenvectors by eigenvalue (variance explained); those are PC1, PC2, ...
Project X onto the top k eigenvectors -> scores matrix with k columns.
Scree plot of eigenvalues shows the natural k (elbow, or cumulative >= ~80-90%).

PCA geometry in 3D.

Data3 features, correlated

PC1axis of max variance

PC2orthogonal, next-most

Projectscore each row on top-k PCs

Hands-on

Examples

Scree and explained variance

import numpy as np
import pandas as pd
from sklearn.datasets import load_digits
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
 
digits = load_digits()
X = StandardScaler().fit_transform(digits.data)
pca = PCA().fit(X)
 
evr = pca.explained_variance_ratio_
cum = np.cumsum(evr)
k_90 = int(np.argmax(cum >= 0.90)) + 1
 
print("n_features:", X.shape[1])
print("PC1-PC5 variances:", np.round(evr[:5], 4))
print(f"components for 90% variance: {k_90}")
Xs = PCA(n_components=2).fit_transform(X)
print("2D embed shape:", Xs.shape)

64 pixel features condense to 2-3 dozen components for most signal and two components for a glance — the scree/total-variance math decides how many to keep.

Real world

In production

Image/region scores: PCA over 100s of census features -> first 2-3 PCs become 'socioeconomic index' style scores.
Preprocessing: decorrelated PCs feed linear models where multicollinearity plagued raw columns.
Embedding inputs: PCA-quantized gesture/telemetry frames compress before an anomaly detector.

Guidelines

Best practices

Standardize before PCA; choose k from cumulative variance AND downstream score, not scree alone; make the projection SVD-based for numerical stability; remember PCs are linear and direction-free — flips are fine; log the loadings for audits.

Watch out

Common mistakes

Forgetting to scale (units drive components); assuming the top PC is 'the important variable' (it mixes many); PCA on sparse count matrices (use TruncatedSVD); treating PCA-transformed columns as causes.

When things break

Debugging tips

Cumulative-variance curve flat before 90%? You have many weak dimensions — likely noise-heavy data.
Compare downstream model score on raw vs projected features; if projection loses nothing, keep it.
Sanity: reproject a row and reconstruct (inverse_transform) — reconstruction error is the compression loss.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

Eigenvalue of a principal component equals:

Before PCA you should usually:

Go deeper

Official documentation

PCA (sklearn decomposition)

docs

Principal component analysis | Wikipedia