Statistics
Intermediate 14 min read Statistics

Regression Basics

Modeling relationships, not proving them

Linear regression models a target as a linear function of predictors, giving coefficients, R² and the machinery for forecasting and adjustment — including the critical caveat that coefficients are not causal without design.

linear regressioncoefficientsR-squaredleast squaresinterpretabilityOLS
Prerequisites: Correlation vs Causation
Before we start

What you will learn

  • Fit OLS regression and read coefficients
  • Understand R², residuals and their limits
  • Interpret relationships as associative unless designed causally
Introduction

What is this?

Linear regression fits y = β0 + β1·x1 + … + ε by least squares. Coefficients βk estimate how much y changes per unit of xk, holding the other predictors' linear effects fixed. R² states the share of variance explained. It is the workhorse for forecasting, adjustment and exploratory hypotheses — but the coefficients describe association, and causal claims need experiment or quasi-experimental design.

Motivation

Why do you need it?

  • Nearly every quantitative discipline's models build from regression reasoning.
  • Forecasting continuous outcomes (revenue, demand, costs) starts here.
  • Adjusting for confounders ('controlling for plan outcome') is regression's day job.
  • Its interpretability beats black-box models for decision reports.
Think about it

Mental model

Imagine

A zoomable map with a route overlay

Think of

Regression

Regression draws the straight path the data tends to follow (fitted line) and reports the grade (slope). Zoom out — the map says 'where it generally goes', not 'why the road bends' at any particular spot.

Mechanics

How it works

  • Ordinary Least Squares minimizes the sum of squared vertical errors.
  • β1 is the slope: expected unit change in y per unit change in x, all else (also) in the model held.
  • R² = 1 − (residual variance / total variance); high R² shows strong linear fit, not causality.
  • Residuals (observed − fitted) should scatter without pattern; patterns reveal poor fit.
  • Multicollinearity between predictors makes individual coefficients unstable.

OLS geometry.

Scatter
Fitted line
Residuals
New x
Hands-on

Examples

Regression in statsmodels

Regression in statsmodels
import numpy as np, pandas as pd
import statsmodels.api as sm
np.random.seed(6)
n = 200
ads = np.random.normal(50, 15, n)
sales = 20 + 1.8*ads + np.random.normal(0, 10, n)
df = pd.DataFrame({"ads": ads, "sales": sales})
df = sm.add_constant(df)
model = sm.OLS(df["sales"], df[["const", "ads"]]).fit()
print(round(model.params["ads"], 2))     # ≈1.8
print("R2=%.3f" % model.rsquared)

The fitted slope recovers the true 1.8 with noise; R² near-high shows the linear fit captures the relationship — nothing causal claimed.

Real world

In production

  • Forecasting next-month demand from price and marketing spend (associative, for planning).
  • Adjusting revenue comparisons for seasonality and company size.
  • Exploratory covariate screening before designing an experiment.
Guidelines

Best practices

  • Check residuals for patterns, not just R².
  • Report coefficients with CIs, not just point estimates.
  • Keep predictors few and meaningful; avoid kitchen-sink models without domain reason.
  • Never frame regression output as causal without an experimental design.
Watch out

Common mistakes

  • Calling the slope 'the effect of ads on sales' when ads correlate with other tactics.
  • Chasing R² and overfitting noise.
  • Ignoring multicollinearity and then trusting unstable coefficients.
  • Extrapolating the fitted line far beyond the observed data range.
When things break

Debugging tips

  • Slope opposite of intuition? Check for multicollinearity or a confounding driver first.
  • R² huge and implausible — check for accidental leakage (predictor derived from target).
  • Fitted values negative for a positive-only outcome — try a generalized linear model (log link).
  • Residual fan shapes — heteroscedasticity; use robust standard errors.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The R² of a regression tells you:

2

A high R² regression is strong evidence the predictor causes the outcome.

Go deeper

Official documentation

statsmodels OLS docs
docs