Regression Diagnostics
Believe the model only after checking its residues
Residual plots, leverage, influence and normality checks reveal whether a regression is honest — before you believe its coefficients or its R2.
What you will learn
- Read residual-vs-fitted and Q-Q plots
- Detect heteroscedasticity and non-linearity from residual patterns
- Spot high-leverage/high-influence points that silently drive the fit
What is this?
Regression diagnostics are the checks that validate every assumption before a regression is trusted: residuals should scatter with no pattern (linearity + homoscedasticity), approximate normality (for inference, not necessarily prediction), and no single row should dominate the fit. Tools: residual-vs-fitted plots, Q-Q charts, leverage/influence statistics (Hat values, Cook’s distance).
Why do you need it?
- A beautiful R2 with a broken residual plot is a beautiful lie used to justify decisions.
- One influential row can flip coefficients — diagnostics catch the 'typo that becomes a strategy'.
- Heteroscedasticity and curvature change which fixes (transformations, weights, model family) to apply.
- Regulated industries audit residual behavior before signing off a model.
Mental model
A doctor reading a full panel, not just the summary score
Regression diagnostics
The 'cholesterol number' (R2) can look fine while the detailed curve (residuals) shows a deeper problem. Numbers summarize, plots diagnose.
How it works
- Residual-vs-fitted: random horizontal band = good; U-shape = curvature; funnel = heteroscedasticity.
- Q-Q plot of residuals: straight-ish line = roughly normal; S-shape/tails = skewed residuals.
- Leverage: unusually extreme x; influence: changes fit when removed (Cook’s distance > ~4/n attention).
- Try fixes in order: curvature -> polynomial/log; funnel -> weighted LS or log y; influential row -> verify data, don't blindly drop.
- Re-run after every fix until plots are clean.
Residual pattern dictionary.
Examples
Diagnostics on a healthy vs broken fit
import numpy as npimport statsmodels.api as sm rng = np.random.RandomState(0)x = rng.uniform(1, 20, 300)y_ok = 0.8 * x + rng.normal(0, 2, 300)y_curvy = np.sin(x) + x * 0.3 + rng.normal(0, 2, 300) model_ok = sm.OLS(y_ok, sm.add_constant(x)).fit()model_c = sm.OLS(y_curvy, sm.add_constant(x)).fit() print("healthy: R2", round(model_ok.rsquared, 3), "resid corr with x^2", round(np.corrcoef(model_ok.resid, x**2)[0,1], 3))print("curvy: R2", round(model_c.rsquared, 3), "resid corr with x^2", round(np.corrcoef(model_c.resid, x**2)[0,1], 3)) import numpy.linalg as lainfl = model_ok.get_influence()print("max Cook's D:", round(infl.cooks_distance[0].max(), 3))R2 looks decent for the curvy fit, but the residual correlation with x^2 reveals structure the model missed — and Cook's distance flags rows that could silently steer the fit.
In production
- Audit-time model reviews: residual plots are the first artifacts the risk team asks for.
- Bug-finding: a funnel or clump often traces back to a bad join or unit mismatch.
- Coefficient confidence: leverage checks protect 'the VP's favorite number' from one wild row.
Best practices
- Make residual plots a standing step in the fit checklist; compute Cook's distance before trusting any coefficient; transform with a documented reason (not to 'fit plots'); keep diagnostics in the model card.
Common mistakes
- Interpreting a Q-Q deviation as fatal for PREDICTIVE use (normality matters for inference, less for fit); deleting influential points without data reasoning; believing R2 one staff at a time; running diagnostics only on the training set.
Debugging tips
- Curved residual band with good R2 => hidden structure (polynomial/log terms).
- A few huge residuals => check those rows for data errors before modeling.
- Funnel => scale the target (log) or move to WLS.
Interview checkpoint
Quick quiz
A U-shaped residual-fitted plot means:
Cook's distance flags: