ML Foundations
Intermediate 14 min read ML Foundations

Regression & Classification

Predicting numbers vs deciding categories

Two canonical ML tasks: predict a number (regression) or predict a category (classification). Understanding their differences, loss functions and evaluation gates is the base layer for all supervised AI work.

regressionclassificationsupervised learninglosslogistic regression
Prerequisites: Train/Test Splits
Before we start

What you will learn

  • Choose regression vs classification from the output type
  • Explain the loss functions each optimizes
  • Pick evaluation metrics that match the business decision
Introduction

What is this?

Supervised ML splits into two workhorses. Regression predicts a continuous number — price, latency, load. Classification assigns one of a set of categories — spam/not-spam, sentiment, intent. The architecture differs (last-layer shape and loss), but the pipeline is identical; the defining difference is the output type and the metric that judges it.

Motivation

Why do you need it?

  • Mixing them up silently (classifying a price, regressing a category) produces nonsense metrics.
  • Regression optimizes distance (MSE/MAE) — sensitive to outliers; classification optimizes a probability/decision.
  • The metric you choose changes the model: MAE vs RMSE favors different errors.
  • Most ML bugs in production trace to the wrong task framing, not the math.
Think about it

Mental model

Imagine

A scale vs a menu

Think of

Regression vs classification

A scale returns a continuous weight (regression); a menu returns one of a fixed set of dishes (classification). Same ingredients, different output — you measure each with different tools.

Mechanics

How it works

  • Regression models output a real number; the loss (MSE, MAE, Huber) penalizes distance to the true target.
  • Classification models output logits → softmax probabilities over classes; cross-entropy loss trains them.
  • Binary classification is a special case: one probability, threshold to decide the class.
  • Feature pipelines are usually identical; only the head, loss and metric change.
  • Evaluation: regression uses R²/MAE/RMSE; classification uses accuracy, precision/recall, ROC-AUC.

From input to number or class.

Features
Head
Loss
Output
Judge
Hands-on

Examples

Framing the task changes everything

Framing the task changes everything
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.metrics import mean_absolute_error, f1_score
 
# regression: predict a price
reg = LinearRegression().fit(X_train, y_price_train)
# classification: predict buy vs not
clf = LogisticRegression().fit(X_train, y_buy_train)
 
print(mean_absolute_error(y_test, reg.predict(X_test)))
print(f1_score(y_test, clf.predict(X_test)))

The same features feed two different heads and two different metrics; framing the target correctly is the whole game.

Real world

In production

  • Cost projection regresses next-month spend; a holding-call classifier decides 'will churn'.
  • Content safety thresholds a toxicity probability at the precision point the product demands.
  • Recommendation re-ranking regresses engagement score; then a threshold decides what's shown.
Guidelines

Best practices

  • Frame by output: continuous value → regression; discrete set → classification.
  • Optimize the loss the metric cares about (MAE for robust, MSE for heavy-outlier).
  • For classification, threshold on cost — tune the cut for precision/recall balance, not default 0.5.
  • Use calibration (probability → honest confidence) if downstream takes probabilities.
Watch out

Common mistakes

  • Regressing a count or category and calling a rounded value a 'class'.
  • Choosing accuracy on an imbalanced classifier and declaring victory.
  • Judging a regression with classification metrics and vice versa.
  • Ignoring outliers when optimizing MSE — a few big errors dominate the fit.
Root cause analysis

Common issues

Error
Regression metrics terrible on a clean-looking task
Root cause

Outliers dominating the loss or wrong encoding

Fix

Use Huber/MAE or clean the targets; inspect residual plots.

Error
Classifier over-confident on every input
Root cause

Feature leakage or an unbalanced training signal

Fix

Check calibration and inspect which features drive the logits.

Error
Great offline AUC, bad in production
Root cause

Distribution shift between eval and serving distributions

Fix

Monitor feature drift and retrain the decision threshold on serving data.

When things break

Debugging tips

  • A 'classifier' outputting fractional values — you built a regressor under a class name.
  • Probability all at 0.999 — calibration or feature leakage, check logits and inputs.
  • Compare MAE vs RMSE; a wide gap flags outlier-driven errors.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which task predicts a continuous number?

2

Cross-entropy is the canonical loss for classification.

Go deeper

Official documentation

scikit-learn regression models
docs
scikit-learn metrics
docs