Regression
Intermediate 12 min read Regression

Polynomial Regression

Curves with linear machinery

Polynomial regression adds powers of features (x, x^2, x^3, interactions) to capture curvature, while remaining a LINEAR model in its parameters — controllable, but quick to overfit with high degrees.

polynomial regressiondegreecurvatureinteraction termsPolynomialFeaturesoverfit
Prerequisites: Multiple Linear Regression
Before we start

What you will learn

  • Explain why polynomial terms are a linear model in parameters
  • Fit PolynomialFeatures + LinearRegression and sweep degree
  • Diagnose low-degree underfit vs high-degree overfit via validation
Introduction

What is this?

Polynomial regression augments features with powers and cross-products ​x, x^2, x^3, x·z..., then fits OLS on the expanded matrix. It remains linear in parameters w (coefficients), so all linear toolbox machinery applies — including ridge (important for high degrees). The degree controls curvature capacity, and with it the classic bias-variance curve.

Motivation

Why do you need it?

  • The real world curves; polynomials are the tractable way to capture smooth curvature.
  • Reuses every linear-model guarantee: closed-form fits, diagnostics, regularization.
  • Ridge on polynomial features is the piece of engineering that tames high-degree fits.
  • It builds the intuition for splines and kernels (both are basis-expansion ideas).
Think about it

Mental model

Imagine

Tracing a coaster's curve with ruler-segmented lines

Think of

Higher-degree polynomial fits

A 1-line ruler (degree 1) misses the loop; more ruler segments (degree 8) follow it but wobble wildly between points. Beauty in fitting = enough segments, not more segments.

Mechanics

How it works

  • Expand X -> [1, x, x^2, ..., x^d] (PolynomialFeatures handles powers and interactions).
  • Fit OLS on the expanded matrix — the unknowns stay the linear coefficients.
  • Sweep degree on validation; watch train/valid curves for the overfit crossover.
  • Add ridge penalty at high degrees; standardize terms (they span disparate ranges).
  • Stay in-bounds: extrapolating a d>=4 polynomial past the observed range explodes violently.

Degree sweep on the bias-variance axis.

d=1
d=3-4
d=10+
Ridge
Hands-on

Examples

Degree sweep with train/test error

Degree sweep with train/test error
import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
 
rng = np.random.RandomState(0)
x = np.linspace(0, 10, 250)
y = np.sin(x) + 0.2 * x + rng.normal(0, 0.4, 250)
Xtr, Xte, ytr, yte = train_test_split(x.reshape(-1, 1), y, test_size=0.3, random_state=1)
 
for d in [1, 3, 7, 15]:
    m = make_pipeline(PolynomialFeatures(d), LinearRegression()).fit(Xtr, ytr)
    print(f"degree {d:>2}: train {mean_squared_error(ytr, m.predict(Xtr)):.3f}"
          f"  test {mean_squared_error(yte, m.predict(Xte)):.3f}")

Low degree underfits the wave; degree ~7 fits it; degree 15 chases noise — the validation curve picks the crossover.

Real world

In production

  • Physical/engineered curves: power laws and saturation curves map naturally to low-degree polynomials.
  • Response surfaces: polynomial fits to A/B point data sketch elasticity shapes cheaply.
  • Spline-lite baselines: piecewise biscuits — especially with ridge — beat plain OLS on curvature-heavy revenue data.
Guidelines

Best practices

  • Keep degree modest (<=4 for inference) unless ridge is on; standardize polynomial features; evaluate with CV, not peak train fit; report extrapolation limits to stakeholders explicitly.
Watch out

Common mistakes

  • Running degree 15 without regularization then blaming 'polynomials'; extrapolating wildly; forgetting interaction terms when they genuinely exist; reading coefficient magnitude of x^d directly as 'importance'.
When things break

Debugging tips

  • Validation error jumps while train keeps falling = classic overfit crossover at high degree.
  • Check feature ranges after expansion — standardization is mandatory for ridge to behave.
  • Use a long-held test range to prove extrapolation blow-up before it bites in prod.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Why ridge on polynomial features helps:

2

Degree 15 on 250 rows almost certainly:

Go deeper

Official documentation

PolynomialFeatures (sklearn)
docs
Polynomial regression | Wikipedia
docs
Numpy polyfit walkthrough
docs