Regression
Intermediate 13 min read Regression

Multiple Linear Regression

Many predictors, one honest readout

Multiple regression learns y = w1x1 + ... + wpxp + b over many features simultaneously, with coefficients read as partial effects 'holding the rest fixed'.

multiple regressionpartial effectsmulticollinearityfeature matrixcoefficients
Prerequisites: Linear Regression
Before we start

What you will learn

  • Fit multivariate least squares and interpret each coefficient conditionally
  • Spot multicollinearity and its effect on coefficient stability
  • Plan feature scaling so coefficient magnitudes stay comparable
Introduction

What is this?

Multiple linear regression extends y = w·x + b to p predictors. Coefficients are partial effects: w_j is the expected change in y for a unit change in x_j holding the other features constant. Near-collinear features war the solution — coefficients explode or flip — which raw OLS accepts silently and ridge penalizes.

Motivation

Why do you need it?

  • Real problems have many simultaneous drivers; single-feature lines lie about confounders.
  • Partial-effect coefficients are the most defensible way to explain a regression to non-ML audiences.
  • Multicollinearity, scaling and residual analysis all live in the multi-feature world.
  • It's the direct foundation for ridge/lasso and every linear-headed model downstream.
Think about it

Mental model

Imagine

A recipe's spice multipliers (add this, hold others fixed)

Think of

Partial-effect coefficients

Each spice's coefficient is its marginal contribution with the rest of the recipe held constant — not its standalone effect if everything else changed.

Mechanics

How it works

  • Assemble the feature matrix X (n x p) with an intercept column.
  • Skip naive matrix inversion; use SVD-based least squares for numerical stability.
  • Coefficient w_j = partial effect; standard error/CI tell significance (use statsmodels for inference).
  • Check variance inflation: VIF > ~10 flags a collinear feature worth dropping or penalizing.
  • Residual checks now run per-feature AND globally (Q-Q plots, residual scatter vs each x).

Multi-feature plane fit.

Features
Estimate
Partial
Check
Hands-on

Examples

Conditional coefficients and VIF

Conditional coefficients and VIF
import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
 
rng = np.random.RandomState(0)
n = 300
x1 = rng.normal(0, 1, n)
x2 = rng.normal(0, 1, n)
x3 = x1 * 0.98 + rng.normal(0, 0.05, n)   # collinear with x1
y = 2 * x1 + 3 * x2 + rng.normal(0, 1, n)
 
m = LinearRegression().fit(np.column_stack([x1, x2, x3]), y)
print("coefs [x1, x2, x3]:", np.round(m.coef_, 3))
 
# VIF quick-check
R = np.corrcoef([x1, x2, x3])
import numpy.linalg as la
VIF = [1 / (1 - R[i, np.arange(3) != i] @ la.inv(R[np.ix_(np.arange(3) != i, np.arange(3) != i)]) @ R[i, np.arange(3) != i].T) for i in range(3)]
print("VIF [x1, x2, x3]:", np.round(VIF, 2))

x1 and x3 are near-duplicates: their coefficients shrink/flip as OLS splits shared signal between them, while VIF shoots up — the red flag that triggers penalization.

Real world

In production

  • Pricing models: price, promo, season coefficients give the demand elasticity per driver.
  • Ops capacity: headcount, task mix and automation coefficients feed planning decisions.
  • Attribution (with caveats): coefficient signs help explain which features drag metrics up or down.
Guidelines

Best practices

  • Standardize features for comparable coefficients; run VIF before trusting; report confidence intervals when inference matters; keep interaction terms explicit and deliberate.
Watch out

Common mistakes

  • Interpreting coefficients when features are collinear (they're unstable); scaling after examining VIF; adding every feature and reading the t-statistics as independent evidence; forgetting an intercept in plain sklearn (it adds one implicitly).
When things break

Debugging tips

  • Sign flips between single-feature and multiple regression = collinearity; check VIF.
  • Coefficient magnitude blow-ups typically flag standardization missed or near-collinearity.
  • Residual plot vs each feature separately surfaces per-feature curvature.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

A VIF above ~10 signals:

2

Partial effect means:

Go deeper

Official documentation

LinearRegression with multiple features (sklearn)
docs
Multicollinearity | Wikipedia
docs
Multiple linear regression explained (video)
video