Mathematics
Intermediate 15 min read Mathematics

Matrix Inverses and Rank

Why some learning problems have no unique answer

The inverse of a matrix solves linear systems like X·w = y, and rank tells you when a solution is unique, infinite or impossible — the root of the feature-collinearity warnings in regression.

inverseranksingular matrixcollinearitylinear systemspseudoinverse
Prerequisites: Matrices and Multiplication
Before we start

What you will learn

  • Understand what A⁻¹ means geometrically (undo the transform)
  • Detect singular / non-invertible matrices via rank and determinant
  • Connect collinearity in features to a rank-deficient design matrix
Introduction

What is this?

Given a square matrix A, its inverse A⁻¹ is the matrix that undoes A's transformation, so A⁻¹A = I. Solving X·w = y — the normal-equation path of linear regression — is done via inversion or the pseudoinverse. If A has no inverse it is singular: its columns are linearly dependent (redundant), and the system has infinitely many or no solutions.

Motivation

Why do you need it?

  • Linear regression's closed-form solution is w = (XᵀX)⁻¹Xᵀy — inverse inside.
  • When features are collinear, XᵀX is singular and the formula explodes with huge weights.
  • Rank tells you how much of the data is genuinely independent information.
  • Regularization (ridge) exists precisely to stabilize this inversion.
Think about it

Mental model

Imagine

Trying to undo 'multiply by zero'

Think of

Inverting a singular matrix

Multiplying a number by zero loses all information — you can't undo it. A singular matrix collapses dimensions (squashes 3D into 2D), so information is gone and no inverse exists.

Mechanics

How it works

  • A square n×n matrix is invertible if and only if its n columns are linearly independent — no column is a combination of the others.
  • Linearly dependent columns collapse at least one dimension (rank < n), making the transformation non-reversible.
  • The rank of A is the number of independent rows/columns; rank equals n means full rank ⇒ invertible.
  • The determinant is an indirect test: det(A) = 0 ⇒ singular, det(A) ≠ 0 ⇒ invertible.
  • For rectangular or singular systems, the pseudoinverse A⁺ (Moore–Penrose) gives the least-squares solution — even when no exact solve exists.

Full rank vs rank-deficient systems.

Full rank A
A⁻¹ exists
Singular A
No inverse
Regularize or pseudoinverse
Hands-on

Examples

Detecting and fixing a singular design matrix

Detecting and fixing a singular design matrix
import numpy as np
X = np.array([[1, 2], [2, 4]])        # column 2 = 2 * column 1
print(np.linalg.det(X))               # ~0.0  → singular
print(np.linalg.matrix_rank(X))       # 1     → only one independent column
 
# Ridge stabilizes the inversion
lam = 1e-3
X_ridge = X.T @ X + lam * np.eye(2)
print(np.linalg.det(X_ridge))         # small but non-zero → invertible

The second column being a multiple of the first collapses rank to 1 and the determinant to zero. Adding λI (ridge) to XᵀX makes it invertible — this is literally what L2 regularization does.

Real world

In production

  • A credit-risk team removes one of two collinear features (income vs salary) after seeing rank-deficient warnings in a ridge regression.
  • An MLOps pipeline crashes with LinAlgError: Singular matrix — a sign that engineered features became duplicates.
  • A recommendation model uses the pseudoinverse to fit collaborative filtering with missing interactions.
Guidelines

Best practices

  • Check X for duplicate or proportional columns after one-hot encoding (dummy trap).
  • Prefer ridge over plain least squares whenever features could be correlated.
  • Use np.linalg.matrix_rank on small examples to confirm intuition before trusting library fits.
Watch out

Common mistakes

  • Ignoring the 'singular matrix' warning from solver libraries — it means your features are redundant.
  • One-hot encoding all k categories of a categorical variable and dropping no column, guaranteeing rank deficiency.
  • Inverting huge matrices directly; solving Xw = y with lstsq instead is faster and more stable.
When things break

Debugging tips

  • When fitting blows up to huge weights, compute matrix_rank(XᵀX) — expect deficiency.
  • Print eigenvalues of XᵀX; near-zero eigenvalues flag the collinear directions.
  • Run df.corr() to spot the redundant feature pairs before you ever fit a model.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

A square matrix has rank < n. What must be true?

2

One-hot encoding all 3 levels of a variable into 3 columns guarantees a rank-deficient matrix.

Go deeper

Official documentation

numpy.linalg (docs)
docs
Moore–Penrose inverse
docs