Mathematics
Advanced 15 min read Mathematics

Linear Algebra and Calculus in ML

One algebra, three model families

A unified view: models are parametric functions, data is a matrix, learning is gradient descent on a loss — and the normal equations, PCA and neural nets are the same algebra at different scales.

normal equationsSVDlow-rankleast squaresmatrix calculuscurvature
Prerequisites: Partial Derivatives Eigenvalues
Before we start

What you will learn

  • Derive the normal equations of linear regression
  • Explain the SVD and why it dominates numerical ML
  • Connect matrix calculus to model training and model capacity
Introduction

What is this?

Most of ML sits on one formula: minimize ||Xw − y||². Linear regression solves it in closed form (normal equations); PCA solves a related problem on the covariance matrix; neural networks solve a non-linear version with gradient descent. Numerical linear algebra — especially the singular value decomposition (SVD) — is the engine under all of them, giving the largest gains in stability, speed and insight.

Motivation

Why do you need it?

  • The normal equations w = (XᵀX)⁻¹Xᵀy are one line of code and one line of math — worth deriving once.
  • The SVD decomposes X = UΣVᵀ; it reveals rank, the best low-rank approximation, and underlies PCA, recommendation and compression.
  • Forcing models to low rank (drop small singular values) is a core compression trick.
  • Matrix calculus (∂L/∂X with respect to matrices) is the math autodiff performs internally.
Think about it

Mental model

Imagine

The steps, the trail map and the terrain

Think of

Calculus, linear algebra and the loss landscape

Gradient descent is the steps; eigenvectors/SVD is the map of terrain features; the curvature (Hessian) tells you how the terrain bends. Together they predict how training will behave.

Mechanics

How it works

  • Least squares minimizes ||Xw − y||²; setting its gradient to zero yields the normal equations XᵀXw = Xᵀy.
  • The SVD factors X as UΣVᵀ where Σ holds singular values σ₁ ≥ σ₂ ≥ ... — a universal 'shape' summary of the data.
  • Truncated SVD (keep k singular values) gives the best rank-k approximation of the matrix — the mathematical heart of PCA.
  • The Hessian ∇²L (matrix of second partials) describes local curvature; Newton-like methods and optimal learning rates depend on it.
  • All this runs in production through LAPACK (the linear algebra library behind numpy/scipy/sklearn).

The same algebra across model types.

Linear regression
PCA / SVD
Deep learning
All consult
Servable model
Hands-on

Examples

Closed-form vs SVD-based regression

Closed-form vs SVD-based regression
import numpy as np
from numpy.linalg import lstsq
X = np.random.RandomState(0).normal(size=(100, 3))
y = X @ np.array([1.0, -2.0, 0.5]) + 0.1
 
# Normal equations — fast but numerically fragile for ill-conditioned X
w_neq  = np.linalg.solve(X.T @ X, X.T @ y)
# SVD-based least squares — the robust default
w_lst  = lstsq(X, y, rcond=None)[0]
print(np.allclose(w_neq, w_lst, atol=1e-6))   # True

np.linalg.lstsq actually uses the SVD under the hood, which is more stable than inverting XᵀX when columns are near-dependent. Knowing this tells you which method to reach for in gnarly real-world fits.

Real world

In production

  • A recommendation engine truncates the SVD of the user-item matrix to 100 latent factors — rank-100 collaborative filtering.
  • An ML platform switches from normal-equations fits to lstsq after a collinear feature set produced garbage weights in staging.
  • A compression team stores model weight matrices at low rank (drop small singular values) and measures a 10× size cut with negligible accuracy loss.
Guidelines

Best practices

  • Prefer lstsq or solve over explicit (XᵀX)⁻¹ — fewer flops, more stable.
  • Check the condition number of XᵀX before trusting a closed-form fit.
  • Personally verify SVD truncation against validation metrics before shipping.
Watch out

Common mistakes

  • Forming (XᵀX)⁻¹ explicitly — expensive and numerically awful.
  • Using full-rank math on inherently low-rank data and wondering why weights oscillate.
  • Assuming closed-form and gradient-descent solutions must differ; they converge to the same answer when done right.
When things break

Debugging tips

  • If two libraries give different fits, compare condition numbers — the ill-conditioned one is wrong.
  • Reconstruct X from its SVD (U Σ Vᵀ) and check residual; large residual means you truncated too aggressively.
  • Print singular values σ₁..σₙ; a cliff in the spectrum flags the effective rank.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The best rank-k approximation of a matrix comes from:

2

The Hessian matrix encodes the local curvature of the loss, not just its slope.

Go deeper

Official documentation

MATHEMATICAL NOTEBOOK (mml-book)
docs
numpy.linalg.svd
docs