Intermediate 14 min read ML Fundamentals

Cross-Validation

Every row is validated exactly once

K-fold cross-validation reuses every row as both training and validation, giving a stable, low-variance estimate of generalization for small and medium datasets.

cross-validationk-foldstratifiedTimeSeriesSplitCV scorevalid set

Prerequisites: Bias-Variance Tradeoff

Before we start

What you will learn

Run KFold, StratifiedKFold and TimeSeriesSplit correctly
Explain why CV estimates beat single splits for small data
Avoid overfitting the CV score itself through repeated reuse

Introduction

What is this?

Cross-validation partitions data into k folds, trains on k-1 folds and validates on the held-out fold per round, rotating so every row validates exactly once. It produces k error estimates whose mean and spread describe generalization more reliably than one split — at roughly k times the training cost.

Motivation

Why do you need it?

Small datasets can't afford a big held-out slice; CV uses all rows for training and validation.
It estimates variance in model quality, not just a single number.
Stratified folds preserve class proportions — critical for imbalanced targets.
Time-series builds are protected from future-peeking via TimeSeriesSplit.

Think about it

Mental model

Imagine

A supervisor rotating workers across stations to assess everyone

Think of

K-fold cross-validation

Every worker (row) gets inspected once while the others cover; you learn how each one performs when their colleagues carry the load.

Mechanics

How it works

Shuffle + split into k equal folds (e.g., 5 or 10).
For each fold: train on the other k-1, validate on the fold, record the metric.
Mean of k metrics = CV estimate; std reports shot-to-shot variability.
StratifiedKFold keeps label-class proportions per fold (classification).
TimeSeriesSplit grows the training window forward, validating on unfolding future slices.
Preprocessing fit inside each fold (scaling/encoding per fold) to avoid leakage.

5-fold CV rotation.

F1validate fold 1, train 2-5

F2rotate: validate 2, train 1,3-5

F3validate 3

F4validate 4

F5validate 5, average

Hands-on

Examples

Three splitters side by side

import numpy as np
from sklearn.model_selection import KFold, StratifiedKFold, TimeSeriesSplit, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
 
X, y = make_classification(n_samples=300, n_features=6, n_classes=2,
                           weights=[0.9, 0.1], random_state=0)
 
print("plain KFold class mix per fold:")
for i, (tr, va) in enumerate(KFold(5, shuffle=True, random_state=1).split(X, y)):
    print(f"  fold {i}: valid y=1 rate {y[va].mean():.2f}")
print("StratifiedKFold class mix per fold:")
for i, (tr, va) in enumerate(StratifiedKFold(5, random_state=1).split(X, y)):
    print(f"  fold {i}: valid y=1 rate {y[va].mean():.2f}")
 
scores = cross_val_score(LogisticRegression(), X, y, cv=StratifiedKFold(5), scoring="f1")
print("stratified F1: mean", round(scores.mean(), 3), "std", round(scores.std(), 3))

Plain KFold can draw a fold with zero positives (invalidating F1); StratifiedKFold guarantees each fold mirrors the 90/10 class balance.

Real world

In production

Feature subset selection: CV scores rank candidate feature sets without overfitting a single split.
Time-series demand: TimeSeriesSplit folds mirror how the deployed model sees history-only data.
Model card: the CV mean/std and fold breakdown become the documented accuracy claim.

Guidelines

Best practices

Always stratify for classification; fit preprocessors inside folds; report mean AND std; remember CV is an estimator — selecting over its output still needs a final independent holdout.

Watch out

Common mistakes

Shuffling time series randomly; fitting scalers on full data then cross-validating (leak); treating the best-of-CV model as directly trustworthy without an untouched holdout; k too small (noise) or too large (expense) without justification.

When things break

Debugging tips

If a single fold's score is a wild outlier, inspect what that fold contains — it may expose a data problem, not a model one.
Check that the mean of per-fold preprocessing refits doesn't silently reuse the same fitted objects.
Fix your randomness (random_state) so runs are reproducible and diffs are meaningful.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

In 5-fold CV, each row appears in validation:

TimeSeriesSplit ensures:

Go deeper

Official documentation

Cross-validation: evaluating estimator performance

docs

K-fold cross-validation | Wikipedia