Beginner 12 min read ML Fundamentals

Training, Testing and Validation

Measure generalization, not memory

You hold out a test set (never touched during training), tune on validation, and train on training data — this triple split is the guardrail that keeps 'model quality' honest.

train/test splitvalidation setholdoutgeneralizationtrain_test_splitleakage

Prerequisites: ML Introduction

Before we start

What you will learn

Implement train/validation/test splits with train_test_split
Explain why test must be untouched until final scoring
Recognize data leakage across split boundaries

Introduction

What is this?

A dataset is partitioned so that model quality can be measured out-of-sample. Training data fits the parameters, validation data tunes choices (hyperparameters, feature sets), and the untouched test set is scored once at the end to estimate real-world performance. A rotten split (leakage across boundaries) corrupts that estimate.

Motivation

Why do you need it?

Train accuracy rewards memorization; real value is out-of-sample performance.
Validation iterated until it looks good becomes a second training set — the test set stays the honest referee.
Time-ordered problems need time-based splits; random splits leak the future.
A trustworthy final number is what shareholders, auditors and users actually read.

Think about it

Mental model

Imagine

A teacher's practice quizzes vs the final exam

Think of

Validation vs test

Study with practice quizzes (validate, adjust) until the exam (test) is genuinely unseen. Taking the actual exam repeatedly to study would defeat the purpose.

Mechanics

How it works

Order your data historically: against time-regulated problems always split chronologically.
train ~60-80%: parameters; validation ~10-20%: hyperparameters; test ~10-20%: final.
Preprocess (scaling, imputation, encoding) fit on train only, applied to the rest.
Tune on validation; then score once on test and freeze the result.
For small data, prefer cross-validation over a single holdout — it reuses all rows.

The three-way split.

Trainfit parameters

Validationtune hyperparameters

Testscore once, then freeze

Prodmonitor, retrain on schedule

Hands-on

Examples

Correct chronological split

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
 
df = pd.DataFrame({
    "value": np.random.RandomState(0).normal(10, 2, 1000),
    "day": pd.date_range("2026-01-01", periods=1000, freq="D"),
})
df["target"] = df["value"] * 1.5 + np.random.RandomState(1).normal(0, 0.2, 1000)
 
df = df.sort_values("day").reset_index(drop=True)
train = df.iloc[:800]
valid = df.iloc[800:900]
test = df.iloc[900:]
 
print("train", len(train), "valid", len(valid), "test", len(test))
print("dates monotone?", df["day"].is_monotonic_increasing)
 
# random split for comparison (WRONG for time series but fine for iid):
tr, te = train_test_split(df, test_size=0.2, random_state=7)
print("random split sizes:", len(tr), len(te))

For temporal data the chronological split keeps the future out of training; a random split would let the model see later days during training — a subtle leakage.

Real world

In production

Demand forecasts: rolling-origin splits beat random splits, which flatter results by peeking at the future.
Fraud: random splits mix criminals and victims wrongly; split by time AND user groups.
A/B evaluation: a fixed, well-labeled holdout is shipped to the finance team as 'the' number.

Guidelines

Best practices

Never touch test until the final scoring; fit ALL preprocessing inside the training partition; for time series use TimeSeriesSplit; keep test proportion sized so the metric estimate is stable (a point estimate without CI on small data misleads).

Watch out

Common mistakes

Fitting the scaler on the full data; validating until test looks good (test becomes validation); random-splitting time series; dropping rows before the split (changes distribution); final-score shopping across many models on test.

When things break

Debugging tips

Assert that label distribution in test approximates train.
Log the exact split so results are reproducible.
If test beats validation suspiciously, suspect leakage.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

Test set best practice is:

For forecasting, which split is correct?

Go deeper

Official documentation

sklearn splitter reference

docs

train_test_split API