Supervised Learning
Intermediate 14 min read Supervised Learning

Linear Models for Classification

The calibrated baseline king

Logistic regression separates classes with a linear decision boundary wrapped in a sigmoid, predicting calibrated probabilities — fast, interpretable, and the reigning baseline.

logistic regressionsigmoiddecision boundaryprobabilitieslinear modelbaseline
Prerequisites: Classification vs Regression
Before we start

What you will learn

  • Explain the sigmoid and the linear decision boundary
  • Fit logistic regression and read its coefficients as log-odds
  • Use it as the mandatory baseline before fancier classifiers
Introduction

What is this?

Logistic regression models ​P(y=1 | x) = σ(w·x + b) with the sigmoid σ(z)=1/(1+e^{-z}). The ​w·x + b = 0 hyperplane is the decision boundary; coefficients are log-odds per feature. Trained by minimizing cross-entropy (with optional L1/L2 penalty), it returns well-calibrated probabilities and is the canonical first classifier for most tabular problems.

Motivation

Why do you need it?

  • It establishes the bar every other classifier must beat — most business data is close to linear at the margins.
  • The output is a real probability, calibratable and auditable.
  • Coefficients are human-readable: effect sizes and directions per feature.
  • Training is milliseconds, robust to collinearity with ridge penalty, and handles missing strategies cleanly.
Think about it

Mental model

Imagine

A balance scale deciding on weight of evidence

Think of

The linear decision boundary

Each feature tips the scale by a fixed amount; the total decides which side wins. How far past the tipping point predicts how confident the call is.

Mechanics

How it works

  • Standardize features: linear models converge faster and coefficients become comparable.
  • Compute the logit z = w·x + b — a linear combination of features.
  • Apply sigmoid: σ(z) compresses z to a probability in (0,1).
  • Train by minimizing cross-entropy; add penalty C (smaller C = stronger regularization).
  • Classify by thresholding the probability at a business-chosen cutoff (not necessarily 0.5).

Sigmoid mapping log-odds to probability.

Logit z
Sigmoid
Prob
Threshold
Hands-on

Examples

Coefficients as log-odds

Coefficients as log-odds
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
 
X, y = make_classification(n_samples=600, n_features=4, n_informative=3,
                           n_redundant=0, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=1)
 
m = LogisticRegression(penalty="l2", C=1.0).fit(Xtr, ytr)
print("AUC:", round(roc_auc_score(yte, m.predict_proba(Xte)[:, 1]), 4))
print("coef (log-odds/unit):", np.round(m.coef_[0], 3))
print("intercept:", round(m.intercept_[0], 3))
 
probs = m.predict_proba(Xte[:3])[:, 1]
print("sample probs:", np.round(probs, 3),
      "-> log-odds:", np.round(np.log(probs / (1 - probs)), 3))

Each coefficient states how many log-odds one unit of that feature contributes; the sigmoid converts total log-odds into a probability — and the two are interchangeable via ​logit.

Real world

In production

  • Credit underwriting: logistic default-probability as the audited scorecard.
  • Marketing churn: calibrated probabilities feed expected-value calculations.
  • Healthcare triage: transparent, small-coefficient models pass regulatory review.
Guidelines

Best practices

  • Scale features before fitting; regularize (C) to control variance; calibrate probabilities if downstream uses them absolutely; check coefficient stability across folds; start here for literally every tabular classification problem.
Watch out

Common mistakes

  • Not standardizing features; leaving the threshold at 0.5 when the business cost is asymmetric; interpreting coefficients as 'effect on probability' (they're log-odds); letting near-collinear features make coefficients wild.
When things break

Debugging tips

  • If coefficients are enormous, scale features or raise penalty.
  • AUC ~0.5 on linear means the boundary is not linear — that's the signal to try trees.
  • Compare calibration curves: a straight 45-degree line means probabilities can be trusted directly.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

What does the sigmoid do?

2

A logistic coefficient of +1.0 means:

Go deeper

Official documentation

LogisticRegression API
docs
Logistic regression tutorial (sklearn walkthrough)
video
Logistic regression | Wikipedia
docs