Beginner 12 min read Regression

Linear Regression

The baseline every regressor must beat

Linear regression fits a line (or plane) y = wx + b to minimize squared-error loss — the workhorse baseline for every continuous prediction problem.

linear regressionordinary least squaresOLSslopeinterceptR2residuals

Prerequisites: Pandas Introduction

Before we start

What you will learn

Fit y = wx + b with Ordinary Least Squares and read slope/intercept
Interpret the fitted line and its residual pattern
Know when linear is a fine model and when it will embarrass you

Introduction

What is this?

Simple linear regression models y = w·x + b and solves for the (w, b) that minimize the sum of squared residuals. Closed-form OLS exists for the single- and multi-feature case; the slope tells the per-unit effect and the residuals tell you whether linearity holds. It's fast, interpretable, and the baseline of record.

Motivation

Why do you need it?

Most simple relationships are approximately linear; a strong linear model defeats plenty of complicated ones.
Coefficients are directly actionable: 'each unit of X moves y by w'.
Residual analysis over a fitted line is the fastest honesty check a regressor can run.
It's where regularization (ridge/lasso) and every other regression concept anchor.

Think about it

Mental model

Imagine

The best straight line through a scatter of plots

Think of

Fitting y = wx + b by least squares

No single line passes through everything, so OLS chooses the one whose total squared vertical misses are smallest — the compromise line.

Mechanics

How it works

Given n points (x_i, y_i), minimize sum_i (y_i - (w x_i + b))^2.
Closed form (single x): w = cov(x,y)/var(x), b = mean(y) - w*mean(x).
For many features, solve the normal equations or use SVD-based least squares.
Evaluate with R2 and residuals; residual scatter reveals curvature or variance drift.
Confirm extrapolation caution: linearity holds inside the observed range only.

Least-squares fit.

Pointsscatter of (x, y)

Liney = w.x + b

Residualsvertical gaps

Minimizesum of squared gaps

Hands-on

Examples

Fit, interpret, and residual-check

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
 
rng = np.random.RandomState(0)
x = rng.uniform(0, 10, 200)
y = 2.5 * x + 3 + rng.normal(0, 1.5, 200)
 
X = x.reshape(-1, 1)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=1)
 
m = LinearRegression().fit(Xtr, ytr)
print("slope", round(m.coef_[0], 3), "intercept", round(m.intercept_, 3))
print("R2", round(m.score(Xte, yte), 3), "RMSE",
      round(np.sqrt(np.mean((yte - m.predict(Xte)) ** 2)), 3))

The estimated slope lands near the true 2.5 and R2 near-perfect for this additive noise — the classic first pass every regressor project starts with.

Real world

In production

Pricing: base rate + linear driver effects give a transparent, tunable formula.
Demand: linear demand curves from price experiments estimate elasticities.
Capacity: linear scaling laws for compute/cost per user guide spend decisions.

Guidelines

Best practices

Always plot residuals after fitting; use the train-only fit for evaluation; standardize features when comparing coefficients; prefer ridge/lasso over plain OLS when features are numerous or collinear.

Watch out

Common mistakes

Judging the model before looking at residuals; extrapolating past the observed range; interpreting R2 in 0-100% 'correctness' terms; forgetting that correlation-driven coefficients aren't causal.

When things break

Debugging tips

A U-shaped residual cloud => curvature: add polynomial or log terms.
Funnel-shaped residuals => heteroscedasticity: consider weighted least squares or log y.
Single dominant residual => likely data-entry bug, not a modeling miss.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

A slope of 2.5 in y = 2.5x + 3 means:

U-shaped residuals imply:

Go deeper

Official documentation

LinearRegression (sklearn)

docs

Ordinary least squares | Wikipedia