Classification
Intermediate 13 min read Classification

Multiclass Classification

Beyond two buckets

For three-plus classes: one-vs-rest trains one binary model per class, softmax (multi-class logistic) shares a single decision surface, and trees/GBM handle multiclass natively by adding per-class scores.

multiclassone-vs-restone-vs-onesoftmaxcross-entropy forwardlabel encoding
Prerequisites: Classification Overview
Before we start

What you will learn

  • Describe OVR vs OVO vs softmax strategies
  • Use OneVsRestClassifier / multi_class logistic properly
  • Pick multiclass metrics (macro/micro F1, confusion heatmap)
Introduction

What is this?

Multiclass classification assigns one of K>2 unordered labels. Strategies: One-vs-Rest (K binary classifiers, argmax of their scores/probs), One-vs-One (K(K-1)/2 votes — costlier), softmax multi-class logistic (one weight vector per class, P(class=k|x) = softmax(z_k)), and native multi-output trees. Evaluation: macro-averaged F1 (each class equally) vs micro (pooled counts), plus the K x K confusion matrix.

Motivation

Why do you need it?

  • Products face a menu (retire candidates vs curious vs loyal), not a toggle.
  • The encoding strategy changes interpretation and cost; softmax is modern default; OVR is the interop-preserving fallback.
  • Macro-vs-micro F1 arguments decide how rare classes count — a metric is a policy.
  • Confusion matrices of multiclass models are the finest debugging device in the toolkit.
Think about it

Mental model

Imagine

Auditioning K soloists vs teaching one ensemble

Think of

OVR vs softmax

OVR: each specialist says 'yes/no for me' and the loudest wins. Softmax: one orchestra line-up picks directly with shared probabilities. Both stage the show; one reuses K binaries, the other learns them together.

Mechanics

How it works

  • Label-encode classes first; never leave strings (predict order = label order).
  • OVR: fit K binary (e.g., logistic) models; for each row pick argmax class probability.
  • Softmax logistic: single model, K output scores, normalize to posterior over classes.
  • Trees/forests/GBM: native multiclass via per-class probabilities without strategy wrapping.
  • Evaluate: macro F1 (class fairness), micro F1 (heavy-class fairness), and the confusion heatmap.

Two multiclass routes.

OVR
Softmax
Trees
Evaluate
Hands-on

Examples

OVR vs softmax on letters

OVR vs softmax on letters
import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.multiclass import OneVsRestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import f1_score, confusion_matrix
 
d = load_digits()
Xtr, Xte, ytr, yte = train_test_split(d.data, d.target, test_size=0.3,
                                      stratify=d.target, random_state=1)
 
ovr = OneVsRestClassifier(LogisticRegression(max_iter=5000)).fit(Xtr, ytr)
sm = LogisticRegression(max_iter=5000, multi_class="multinomial").fit(Xtr, ytr)
 
for name, m in [("one-vs-rest", ovr), ("softmax", sm)]:
    p = m.predict(Xte)
    print(f"{name:>11}: macro-F1 {f1_score(yte, p, average='macro'):.4f}"
          f"  accuracy {np.mean(p == yte):.4f}")
 
cm = confusion_matrix(yte, sm.predict(Xte))
print("confusion heatmap diagonal:", np.diag(cm).sum(), "/", cm.sum(),
      "-> classes 3,8,9 hardest:",
      (np.diag(cm) / np.maximum(cm.sum(axis=1), 1))[3:10].round(2))

Both strategies land near-equal scores on digits; the confusion matrix then pinpoints which class pairs the boundary still mixes.

Real world

In production

  • Document ROI classifier: softmax over 'renew/upsell/ignore' with SHAP for the leading class.
  • Incident triage: OVR as model family compatibility when each severity needs its own guardrail.
  • Image classes: softmax-based nets output interpretable posteriors directly.
Guidelines

Best practices

  • Stratify by class in splits; pick macro vs micro from the stakeholder policy BEFORE modeling; with highly imbalanced multiclass, apply per-class weights and report recall per class; log the label-order mapping.
Watch out

Common mistakes

  • One-vs-all with an unbalanced per-class prior (weights fix); measuring only accuracy where a win on 9-of-10 classes hides a lost class; using strings directly as y in metrics; forgetting stratify produces empty-class folds.
When things break

Debugging tips

  • Confusion matrix off-diagonal peaks reveal confusable pairs — usually gendered feature veering.
  • Aggregate per-class recall; a single 0.x class flags an imbalance-and-metric loophole.
  • Check that pipeline labels at predict match training label order (encoder consistency).
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

One-vs-Rest trains:

2

Macro F1 weights classes:

Go deeper

Official documentation

Multiclass and multilabel (sklearn)
docs
Softmax function | Wikipedia
docs
One-vs-all explained (video)
video