Regularization: Ridge and Lasso
Penalize size, trade bias for variance
Ridge (L2) shrinks coefficients smoothly, taming collinearity; Lasso (L1) drives coefficients to exactly zero, doing feature selection; both trade a little bias for much less variance.
What you will learn
- Write ridge and lasso objectives (L2 and L1 penalties)
- Explain why L1 zeroes coefficients while L2 only shrinks them
- Tune alpha via CV and interpret the coefficient path
What is this?
Regularized regression adds a penalty on coefficient magnitude to the squared-error loss: ridge adds α ||w||^2 (L2 — smooth shrinkage), lasso adds α ||w||_1 (L1 — sparsifying, exactly zeroes negligible coefficients). Higher α = more shrinkage, lower variance, higher bias: picking α by CV is the standard recipe for high-dimensional, collinear or noisy feature sets.
Why do you need it?
- Plain OLS explodes on collinear or p>n feature sets; penalization restores a stable solution.
- Lasso automatically prunes features — model simplicity and readability in one fit.
- The bias-variance knob is explicit: α lets you dial exactly where to sit on the curve.
- Every ridge/lasso idea carries over to deep-learning weight decay and L1 feature penalties.
Mental model
Two ways to slim a wardrobe: tailor each piece down vs throw pieces away
Ridge (shrink) vs Lasso (prune)
Ridge keeps every garment but makes each a bit less extreme; Lasso bins pieces entirely. Both reduce the 'noise' of an overpowered closet.
How it works
- Ridge: minimize Σ(y - w·x)^2 + αΣw_j^2 — pull ALL weights toward 0 smoothly.
- Lasso: minimize Σ(y - w·x)^2 + αΣ|w_j| — the L1 geometry (square corners) forces exact zeros.
- ElasticNet mixes both: α(l1 · L1 + (1-l1) · L2) — best of both when groups of collinear features exist.
- Scale features BEFORE penalizing, or the penalty treats units differently.
- Tune α with CV on a log scale; watch the coefficient path as α grows.
L1 vs L2 penalty geometry.
Examples
Lasso on collinear features
import numpy as npfrom sklearn.linear_model import Lasso, Ridgefrom sklearn.model_selection import train_test_split rng = np.random.RandomState(0)n = 300x1 = rng.normal(0, 1, n)x2 = rng.normal(0, 1, n)x3 = x1 * 0.97 + rng.normal(0, 0.06, n) # near-duplicate of x1X = np.column_stack([x1, x2, x3])y = 2 * x1 + 3 * x2 + rng.normal(0, 1, n) ridge = Ridge(alpha=1.0).fit(X, y)lasso = Lasso(alpha=0.05).fit(X, y)print("true w ~ [2, 3, 0]")print("ridge coef:", np.round(ridge.coef_, 3))print("lasso coef:", np.round(lasso.coef_, 3))Ridge splits the shared signal between x1 and its near-duplicate; lasso pushes one of them to zero and recovers a cleaner [2,3,0] readout.
In production
- High-dim health data: lasso picks a handful of laboratory features for a compliant scorecard.
- Marketing mix: lasso simplifies attribution to the few channels that matter.
- Servers with 1k correlated metrics: ridge stabilizes capacity models that OLS can't fit.
Best practices
- Standardize all features before penalizing; tune alpha via CV on log scale; prefer ElasticNet for grouped collinear features; report the selected feature set with lasso (it IS the model card source).
Common mistakes
- Penalizing without standardization; alpha tuned on the test set; expecting lasso to behave uniquely with collinear features (it picks one arbitrarily — elasticnet is safer); reading zeroed coefficients as 'provably irrelevant'.
Debugging tips
- Alpha too high => all coefficients collapse to 0; check the CV curve before it flattens.
- Lasso picks x1 but next fold picks x3: collinearity — switch to ElasticNet or drop one upfront.
- CV-chosen alpha at the search edge: widen the grid.
Interview checkpoint
Quick quiz
Ridge's L2 penalty:
Lasso is often preferred when: