Intermediate 14 min read Supervised Learning

Support Vector Machines

The maximum-margin boundary

SVMs find the maximum-margin separating hyperplane, projecting to high-dim feature spaces via kernels — strong on small-and-medium tabular data, sensitive to scaling and kernel choice.

SVMsupport vectormarginkernelC regularizationhyperplane

Prerequisites: K-Nearest Neighbors

Before we start

What you will learn

Define margin and support vectors in the linear case
Explain what a kernel does and the RBF kernel's two knobs (C, gamma)
Tune SVMs carefully and know their scaling sensitivity

Introduction

What is this?

SVM fits a hyperplane that separates classes with the widest margin: only the closest points (support vectors) define it. When data isn't linearly separable, kernels (polynomial, RBF) implicitly map to a higher-dimensional feature space where separation becomes linear. The two main knobs: C (margin vs misclassification penalty) and gamma (RBF width).

Motivation

Why do you need it?

Max-margin separation generalizes remarkably well for moderate-size data.
Kernels let a rival of trees and linear models handle non-linear boundaries elegantly.
SVMs excel when features are few, well-chosen and scaled, and rows are in the hundreds-to-tens-of-thousands.
Concepts feed directly into modern kernel ideas, dual formulations and embedding geometry.

Think about it

Mental model

Imagine

Drawing the widest possible lane between two crowds

Think of

Finding the maximum-margin hyperplane

Only the people nearest the lane edge (support vectors) define it; everyone far inside the crowd is irrelevant to the drawing. The widest lane is the most robust to jostling (noise).

Mechanics

How it works

Linearly separable: find the hyperplane with the max margin; samples on the margin are support vectors.
Soft margin: C balances margin size against margin violations (misclassified points).
Non-separable: kernel K(x_i, x_j) computes an implicit dot-product in a lifted feature space (RBF: exp(-gamma*d^2)).
small gamma = broad/smooth influence; large gamma = local/narrow, high-variance.
Training solves a convex quadratic-program; prediction depends only on support vectors.

SVM hyperplane with margin.

Pointstwo classes in feature space

Hyperplanew.x + b = 0

Marginwidest gap, defined by SVs

Kernelimplicit non-linear lift

Hands-on

Examples

RBF SVM under C and gamma

import numpy as np
from sklearn.datasets import make_circles
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score
 
X, y = make_circles(n_samples=400, factor=0.5, noise=0.08, random_state=0)
X = StandardScaler().fit_transform(X)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=1)
 
for C, gamma in [(1.0, 0.1), (10, 10), (1e-2, 0.1), (1.0, 100)]:
    m = SVC(C=C, gamma=gamma, probability=True).fit(Xtr, ytr)
    print(f"C={C:<5} gamma={gamma:<5} test acc {accuracy_score(yte, m.predict(Xte)):.3f}"
          f"  n_support {m.n_support_.sum()}")

make_circles is non-linearly separable, so a linear SVM fails but RBF with moderate C/gamma succeeds; radical gamma overfits, tiny C underfits — the two knobs trade margin against fit.

Real world

In production

Document/numeric anomaly screens where small clean labeled sets exist.
Baseline kernels for small tabular biomedical or engineering data (more robust than deep nets at that scale).
SVRegressor for small-sensor regression with strong structure.

Guidelines

Best practices

Standardize every feature; tune C and gamma jointly with randomized/CV search on log scales; avoid SVMs on 100k+ rows (quadratic training cost); keep the feature count small and principled.

Watch out

Common mistakes

Feeding unscaled features; defaulting gamma without search; using SVMs on big-data with deep-feature sets; forgetting probability=True costs extra but gives calibrated-like outputs; expecting speed on millions of rows.

When things break

Debugging tips

Training time exploding with n is the #1 sign SVM is the wrong tool at that scale.
n_support_ equal to n_train means near-memorization — raise C or fix gamma.
Grid over C and gamma on log scale; plot the heatmap for a low-data sanity check.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

A high value of C generally means:

Which dataset runs SVMs badly?

Go deeper

Official documentation

Support Vector Machines (sklearn)

docs

Kernel trick | Wikipedia