Linear Regression Pitfalls
Six ways the dashboard lies
Linear regression fails softly: non-linearity, outliers, multicolinearity, scale, leakage in time data and target skew — each quietly dishonors the number printed on the dashboard.
What you will learn
- Enumerate the classic failure modes of linear models
- Recognize each from symptoms (plot shapes, VIF, train/test divergence)
- Apply the matching fix per failure
What is this?
The pitfalls live in the fit's assumptions: (1) linearity broken (curves), (2) outliers dragging OLS (it minimizes squared error), (3) collinearity destabilizing coefficients, (4) unit-scale asymmetry, (5) target skew distorting error norms, (6) time/data leakage inflating scores. Each is silent until diagnostics or a leak event exposes it.
Why do you need it?
- Linear regression is the default, and defaults drift into production claims.
- Squared-error means ONE outlier can out-weigh a thousand good rows.
- Leakage in time-split data produces dashboard-beautiful models that fail live.
- Knowing the failure catalog earns trust that 'it worked in the notebook' can't.
Mental model
A Swiss knife: right tool for most things, wrong for a plank
Linear regression used universally
The problem isn't the knife — it's sawing a plank with it and reporting 'the blade is broken'. Match the failure to the tool or switch tools.
How it works
- Non-linearity: detect via residual U-shape; fix polynomial/log/GBM.
- Outlier drag: detect via leverage/Cook's; fix robust regression (Huber) or log transforms.
- Collinearity: detect via VIF; fix drop-one / ridge / elasticnet.
- Scale: detect via coefficient magnitudes; fix standardization.
- Target skew: detect via residual histogram/Q-Q; fix log-target (interpret back-transform!).
- Leakage: detect via train/validation divergence or future-feature audit; fix time-splits and pipeline hygiene.
Pitfall -> symptom -> fix table.
Examples
Outlier drag in action
import numpy as npfrom sklearn.linear_model import HuberRegressor, LinearRegression x = np.linspace(0, 10, 100)y = 1.5 * x + np.random.RandomState(0).normal(0, 1, 100)y[95] = 110 # one planted outlier ols = LinearRegression().fit(x.reshape(-1, 1), y)rob = HuberRegressor(epsilon=1.35).fit(x.reshape(-1, 1), y)print("truth: slope 1.5")print("OLS slope:", round(ols.coef_[0], 3))print("Huber slope:", round(rob.coef_[0], 3))One planted point drags OLS's slope off; the Huber-robust fit shrugs it off — proof that a single row can own a plain least-squares answer.
In production
- Forecast dashboards: leaky time splits regularly produce great backtests and live misses — the pitfall audit is part of release.
- Pricing rules derived from OLS: collinear promo flags flip sign weekly; ridge stabilizes the rule.
- Fraud amount models: log-target avoids '$5k typical, $40k mean' distortions.
Best practices
- Run the pitfall checklist after every fit (plots, VIF, leverage, split sanity); time-split anything with date columns; keep the transform documented (log-target means predictions are geometric means — back-transform on deploy!).
Common mistakes
- Back-transform forgetfulness (predicting log(y) and reporting it as y); deleting outliers without evidence; believing a leaky-like validation gap is 'great results'; using R2 on log-scale as if it matched raw-scale meaning.
Debugging tips
- Validation wildly above expectation? Run the leak audit before celebrating.
- Repeatable sign flips on the same feature: collinearity check now.
- Mean far from median on predicted values: back-transformation bug.
Interview checkpoint
Quick quiz
One huge outlier hits OLS hardest because:
Predicting log(y), you must predict y by: