Train, Validation & Test Splits
Honest numbers need clean isolation
To know how a model performs on new data, it must be evaluated on data it never saw in training or tuning. Leak-free, stratified, time-aware splits are the foundation of every honest number.
What you will learn
- Explain the distinct roles of train, validation and test
- Split without leakage: time-aware and grouped
- Use stratification for rare classes
What is this?
The train, validation and test split isolates data by purpose. Training tunes parameters; validation tunes hyperparameters and checks early stopping; the test set is evaluated once at the end, having influenced nothing. Sloppy splits leak: normalization fit on the whole set, random splits on time-series, duplicate rows across sets — all inflate measured performance.
Why do you need it?
- Data that leaks into training teaches the model exam answers instead of the subject.
- Hyperparameters tuned on test contaminates it; test must be final and reserved.
- Random splits on chronological data predict the past with future info — never deployable.
- Stratification guarantees rare classes appear in test so you can judge them.
Mental model
Final exams vs homework
Test vs train data
Teachers grade a separate final exam, never the homework the students practiced on. If you graded from the homework, everyone would be a genius — and life wasn't the homework.
How it works
- A 60-20-20 (or similar) split separates train, validation and test by index or group.
- Validation is used during development: hyperparameter search, early stopping, model selection.
- Test is held out until the final candidate; evaluating it more than decisions corrupts it.
- Time-aware splitting: training on the past, validating on mid periods, testing on the latest window.
- Stratification samples per class so rare classes appear in every split.
Three roles, one reserved set.
In production
- A fraud model trains on months 1-8, validates 9-10 and tests on 11-12 — chronology preserved.
- A rare-disease classifier stratifies by class so the 1% positive class appears in test.
- An eval team freezes the test split at month-boundary and reuses it across experiments to keep apples-to-apples comparisons.
Best practices
- Split before any preprocessing; fit normalizers/encoders on train only.
- Split by group for grouped data (users, stores) so the same entity never straddles sets.
- Keep the test set frozen and almost never look at the real-valued performance on it.
- Report performance distribution, not just a point estimate, on test.
Common mistakes
- Fitting scalers/encoders on the full dataset and calling it 'no leakage'.
- Randomly splitting rows when the entity appears multiple times (same user in both sides).
- Reusing the test set for weekly tuning until it silently becomes validation.
Common issues
Same user appears in train and testRow-level random split on multi-row entities
Split by entity/group id, then distribute their rows wholly into one set.
Baseline 'leaks' the future on developer dataRandom split on chronological data
Time-aware split: train on earlier windows, test on later.
Validation and test diverge wildlyDistribution shift between periods or a small test
Align the split window with the production timeframe and grow test size.
Debugging tips
- If test is suspiciously better than validation, suspect leakage or duplicated rows between the sets.
- Compute max overlap of entity ids across splits to prove grouping held.
- Track which split produced a suspect prediction — a test-only anomaly alerts you to contamination.
Interview checkpoint
Hands-on lab
- Implement a group-aware split so entities never straddle sets
- Implement a time-ordered split
- Detect entity overlap between train and test
Split Without Leaking
25 min · intermediate
Quick quiz
Which set is used to tune hyperparameters?
Fitting your normalizer on the entire dataset before splitting leaks information.