Intermediate 12 min read Supervised Learning

K-Nearest Neighbors

Predict by your closest neighbors

KNN predicts by majority vote (or average) among the k closest training points — lazy, non-parametric, and entirely defined by the distance metric in scaled feature space.

KNNk-nearest neighborsdistanceinstance-basedlazy learningnon-parametric

Prerequisites: Linear Models for Classification

Before we start

What you will learn

Explain lazy/instance-based learning and the role of k
See why scaling is mandatory for distance models
Diagnose k too small (variance) vs k too large (bias)

Introduction

What is this?

K-Nearest Neighbors stores the training set and, at prediction time, finds the k nearest points under a distance metric (Euclidean default), voting their labels (classification) or averaging values (regression). It's non-parametric and lazy — no training phase — but requires scaling so wide features don't own the distance.

Motivation

Why do you need it?

It sketches non-linear boundaries without any training math — a perfect warm-up baseline and intuition engine.
Instance-based prediction shines in small, local datasets and simple recommenders.
Understanding distance geometry transfers directly to clustering, embeddings and retrieval.
It's the standard contrast case: tree/linear/knn each learns from a different notion of 'similar'.

Think about it

Mental model

Imagine

Asking 3 smart friends instead of studying the textbook

Think of

K-nearest neighbors predictions

Lazy learning: nothing internal is built; you consult the k closest advisors (neighbors) each time. Too few advisors = overconfident noise; too many = average personality.

Mechanics

How it works

Scale features (standardize) so no column dominates the Euclidean distance.
Choose k (typical 3-7 for tabular; tune via CV) and distance metric.
For each query point compute distances to all training rows, sort, take the k smallest.
Classification: majority vote among k (ties broken by distance or k+1); regression: mean of the k values.
Larger k = smoother/bias up, variance down; smaller k = ragged/overfit.

k neighbors ball around a query.

Querynew point x

Distancescompute to every row

Sorttake k nearest

Vote / meank labels or values

Hands-on

Examples

k effect on a noisy 2D set

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import f1_score
 
X, y = make_classification(n_samples=700, n_features=6, n_informative=4,
                           n_redundant=1, random_state=0)
Xsc = StandardScaler().fit_transform(X)
Xtr, Xte, ytr, yte = train_test_split(Xsc, y, test_size=0.3, stratify=y, random_state=1)
 
for k in [1, 3, 7, 21, 101]:
    clf = KNeighborsClassifier(n_neighbors=k).fit(Xtr, ytr)
    print(f"k={k:>3}: train F1 {f1_score(ytr, clf.predict(Xtr)):.3f}"
          f"  test F1 {f1_score(yte, clf.predict(Xte)):.3f}")

k=1 memorizes (train perfect, test ragged); growing k smooths and usually lifts test F1 until bias (k=101) drowns the signal.

Real world

In production

Cold-start recommenders: average ratings of k nearest users.
Anomaly pre-screen: deviation from nearest-neighbor density flags outliers cheaply.
Baseline for domains with few features where locality matters (geo, fraud velocity).

Guidelines

Best practices

Standardize before fitting; tune k and metric via CV; for large data use KDTree/ball-tree or subsample; weight by inverse distance for smoother answers; watch that rare classes need enough neighbors present.

Watch out

Common mistakes

Forgetting to scale (income in $ dominates age); k tuned too small so noise drives votes; using KNN on high-dimensional data where Euclidean distance collapses (curse of dimensionality); brute-force O(n) predictions on millions of rows.

When things break

Debugging tips

Compare F1 across k — the curve should be smooth-ish; wild swings mean k too small.
Plot query distances: if most neighbors are nearly equally far, the space is too high-dimensional.
Scale with the SAME scaler fit on train, never refit on live data.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

KNN is called 'lazy' because:

Small k tends to produce:

Go deeper

Official documentation

Nearest Neighbors (sklearn)

docs

Curse of dimensionality | Wikipedia