Mathematics
Intermediate 16 min read Mathematics

Eigenvalues and Eigenvectors

The directions a matrix preserves

An eigenvector is a direction the matrix doesn't rotate, only stretches. Eigenvalues say how much, and they power PCA, spectral clustering, page ranking and dimension reduction.

eigenvectoreigenvaluePCAspectralcovariance matrixdimension reduction
Prerequisites: Matrices and Multiplication
Before we start

What you will learn

  • Define eigenvectors/eigenvalues and interpret them geometrically
  • Compute eigen-decomposition with numpy
  • Relate the covariance matrix's eigenvectors to PCA's principal components
Introduction

What is this?

For a square matrix A, a non-zero vector v is an eigenvector if Av = λv for a scalar λ (the eigenvalue). Geometrically, applying A to v only scales v — it never changes its direction. An eigen-decomposition A = VΛV⁻¹ factors A into its action along these special directions, which is exactly how PCA finds the axes of maximum variance.

Motivation

Why do you need it?

  • PCA computes the eigenvectors of the covariance matrix; the top ones are the principal components.
  • Spectral clustering uses eigenvectors of the graph Laplacian to find communities.
  • PageRank's stationary distribution comes from an eigenvector computation.
  • Many numerical stability properties of matrices are stated via eigenvalues.
Think about it

Mental model

Imagine

A parade going straight through a bending corridor

Think of

An eigenvector through a transformation

Everyone in the parade (the vector) stays in the corridor's direction — only the group's size (the eigenvalue) changes. The matrix bends other directions but lets the eigen-direction pass through unchanged.

Mechanics

How it works

  • Av = λv: multiplying by A keeps v in the same line, scaled by λ.
  • λ > 1 stretches, 0 < λ < 1 shrinks, λ < 0 flips direction, λ = 0 kills the direction (singular).
  • Collecting all eigenvector equations gives A = VΛV⁻¹ where Λ holds eigenvalues on the diagonal.
  • For symmetric matrices (like covariance matrices) eigenvectors are orthonormal — perpendicular unit vectors.
  • PCA keeps the eigenvectors with the largest eigenvalues, because those directions capture the most variance.

From covariance matrix to principal components.

Center data
Covariance matrix
Eigen-decomposition
Sort by λ
Project data
Hands-on

Examples

Hand-roll PCA via eigen-decomposition

Hand-roll PCA via eigen-decomposition
import numpy as np
X = np.random.RandomState(42).normal(size=(200, 3))
C = np.cov(X.T)                      # 3×3 covariance
eigvals, eigvecs = np.linalg.eigh(C) # ascending
print(eigvals)
top = eigvecs[:, -1]                 # largest-eigenvalue direction
# Projected variance along a unit direction u is u^T C u
print(top @ C @ top, np.max(eigvals))

The bulk of the code is one call to np.linalg.eigh. The largest eigenvalue is the variance captured by its eigenvector — the first principal component. sklearn's PCA computes this exact decomposition under the hood.

Real world

In production

  • An image team reduces 4096-pixel features to 100 via the top eigenvectors of the covariance matrix, cutting training time by 30×.
  • A fraud detection system uses the eigen-gap (drop between eigenvalues) to decide how many latent dimensions describe normal behavior.
  • A recommender uses spectral methods on item co-occurrence graphs to find latent topic clusters.
Guidelines

Best practices

  • Center and scale data before PCA, or variance (not correlation) dominates the eigenvectors.
  • Use np.linalg.eigh for symmetric matrices — it's faster and assumes symmetry.
  • Plot the eigenvalue 'elbow' to choose the number of components.
Watch out

Common mistakes

  • Running PCA on raw unscaled features so one big-magnitude column dominates.
  • Forgetting that eigenvalues are sorted ascending in np.linalg.eigh — take the last ones for the top components.
  • Applying PCA to the data matrix directly instead of its covariance when doing it by hand.
When things break

Debugging tips

  • Check that eigenvectors from np.linalg.eigh form an orthonormal set: V @ V.T ≈ I.
  • Reconstruct first: X_hat = X @ V[:, -k:] @ V[:, -k:].T must closely approximate centered X if few components are lost.
  • If PCA 'explains nothing', verify you standardized columns and that variance actually differs across columns.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Av = λv with λ = 2 means:

2

The eigenvector with the largest eigenvalue is the first principal component of PCA.

Go deeper

Official documentation

numpy.linalg.eig extremes
docs
sklearn PCA (docs)
docs