Intermediate 13 min read Classification

Logistic Regression

The calibrated, auditable workhorse

The canonical binary classifier: sigmoid over a linear combination, cross-entropy training, probability output, optional L1/L2 penalties — fast, interpretable, and unbeaten as the default baseline.

logistic regressionsigmoiddecision boundarycross-entropylog-oddsbaseline

Prerequisites: Classification Overview

Before we start

What you will learn

Fit logistic regression and read coefficient log-odds
Use predict_proba and choose thresholds deliberately
Apply penalty/C and standardization for stable coefficients

Introduction

What is this?

Logistic regression models P(y=1|x) = σ(w·x + b) with the sigmoid; trained on binary cross-entropy with an optional L1/L2 penalty (C controls the inverse of regularization strength). It returns probabilities suitable for ranking and thresholds, is trivially deployable, and survives as the strongest simple baseline on most tabular data.

Motivation

Why do you need it?

It's the legal-default classifier: auditable coefficients, milliseconds of training, robust probabilities.
The linear boundary is wrong for some problems — but you need the baseline to PROVE that.
Coefficients map to a story (age, spending, tenure drive logs-odds of churn).
Calibration is generally good out of the box; a Platt/isotonic pass refines it.

Think about it

Mental model

Imagine

A betting shop quoting odds as log-odds

Think of

Logistic coefficients

The shop says '+150 on the underdog' — log-odds. Every feature nudges the quoted odds by a multiplier (coefficient exponent); the sigmoid turns the stacked odds into win probability.

Mechanics

How it works

Compute logit z = w·x + b over all features.
Map to a probability via sigmoid σ(z) = 1/(1+e^{-z}).
Train on cross-entropy: gradient descent nudges w so predicted probabilities match labels.
Standardize features; tune C (smaller = stronger regularization).
Threshold the probability at the cost-optimal cutoff for THE decision.

Logistic decision machinery.

Featuresscaled x

Logitw.x + b

Sigmoid(0,1) probability

Thresholdbusiness cutoff

Hands-on

Examples

Interpret and threshold deliberately

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score
 
X, y = make_classification(n_samples=800, n_features=5, n_informative=4,
                           weights=[0.9, 0.1], random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=1)
 
m = LogisticRegression(max_iter=2000).fit(StandardScaler().fit_transform(Xtr), ytr)
p = m.predict_proba(StandardScaler().fit(Xtr).transform(Xte))[:, 1]
print("AUC:", round(roc_auc_score(yte, p), 4))
print("pred prob at deciles:", np.round(np.quantile(p, [0.1, 0.5, 0.9]), 3))

A single scaler+logistic pipeline produces a ranked list of probabilities; the deciles show where a business-prioritized threshold would naturally sit.

Real world

In production

Credit risk scorecards: regulated, coefficient-explained default probabilities.
Churn propensity: probability feeds the retention queue and expected-value calc.
Fraud prefilter: cheap logistic gate before expensive deep models only on risky rows.

Guidelines

Best practices

Standardize features; fit penalty-C with CV; verify probability calibration (calibration_curve); keep the threshold decision separate from the fit and recorded in the model card.

Watch out

Common mistakes

Interpreting coefficients as probability units (they're log-odds); skipping standardization so coefficients are meaningless; defaulting the threshold to 0.5; trusting probabilities on wildly shifted serving distributions.

When things break

Debugging tips

Coefficient sign flips between simple and full fit => collinearity; check VIF.
AUC ~0.5 after lots of work => boundary is non-linear — the baseline did its job.
Calibration curve bowed => apply isotonic/Platt calibration before using probabilities.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

coef = 1.0 on a standardized feature means:

Cross-entropy loss rewards:

Go deeper

Official documentation

LogisticRegression (sklearn)

docs

Logistic regression | Wikipedia