K-Fold Cross Validation
The workhorse estimator of generalization
K-fold is the workhorse CV strategy: k train/validate re-uses of the data with full coverage. Getting the fold count, seeds and error handling right is what makes it trustworthy.
What you will learn
- Run k-fold correctly and reproducibly
- Choose k by bias-variance and compute trade-offs
- Diagnose unstable folds
What is this?
K-fold cross validation partitions data into k folds, trains on k−1 and validates on the remaining, rotating until each fold was validation once. k=5 and k=10 are the defaults not because they're magical but because they sit at the sweet spot: low fold-to-fold variance with acceptable compute and only mild bias from overlapping training sets.
Why do you need it?
- It produces the most reliable generic performance estimate for modest data sizes.
- It reuses every training row across folds — crucial when data is scarce.
- It gives a per-fold distribution so you can spot fragility the mean hides.
- It's the standard baseline CV in sklearn, MLflow and most platforms.
Mental model
A teacher grading with several question banks
k-fold on your data
Fresh banks (folds) hammer home the same material each round; if the student only ever aces one bank, the other exams expose it. Averaging those exams is a fairer picture than any single bank.
How it works
- Split indices into k contiguous (shuffled) groups via KFold(shuffle=True, random_state=R).
- Loop fold in range(k): train on all others, predict on fold, record metric.
- Aggregate: mean and standard deviation of fold metrics.
- Never reuse the same data as the final report test set; and keep the fold seed versioned.
- Watch compute: k=10 trains 10 models — for deep nets, use fewer folds or sub-sampling.
Rotating validation across 5 folds.
In production
- Baseline model selection across a model zoo is scored by 5-fold CV; the winner then gets a single untouched test evaluation.
- Feature selection loops wrap k-fold so expected loss is computed on out-of-fold predictions.
- An automated training pipeline logs per-fold metrics to the experiment tracker and fails on a wide fold spread.
Best practices
- Shuffle before folding unless the order is meaningful (time, groups).
- Fix and record the random seed — CV is not meant to be a fresh coin-flip each run.
- Report fold mean and spread; 'CV 0.87 ± 0.09' is honest, 'CV 0.87' is not.
- For unbalanced classifiers, prefer StratifiedKFold so each fold mirrors class proportions.
Common mistakes
- Folding without shuffle when the data is sorted by target — one fold becomes all classes, one none.
- Reusing the same k=10 runs to make hundreds of model decisions and believing each is fresh.
- Using KFold on time-series or grouped data without a structure-aware splitter.
- Forgetting to set shuffle before random_state (KFold(shuffle=False) ignores the seed).
Common issues
One fold's metric is drastically lowerGroup structure or class imbalance concentrated into a single fold
Use StratifiedKFold / GroupKFold, or shuffle with a larger k to dilute any single fold's dominance.
Reproducibility: same code, different CV numbersUnseeded splitter or model randomness
Set random_state on the splitter and any stochastic estimator, and log it.
Debugging tips
- Print the class distribution per fold; a fold with no positives poisons its metric.
- If fold metrics are bimodal, inspect which folds those are and whether a rare group dominates them.
- Reproduce a CV run exactly by pinning both the splitter seed and any model-internal seeds.
- Compare k=5 vs k=10 estimates; if they differ hugely, your sample is too small or unstable for either.
Interview checkpoint
Hands-on lab
- Implement 3-fold CV manually over a toy dataset
- Confirm results against sklearn cross_val_score
- Observe how stratified + seeded folds stabilize estimates
Rerun Everything Yourself With K-Fold
30 min · intermediate
Quick quiz
With KFold on data sorted by label, the model tends to:
Fixing the cross-validation seed makes model selection more reproducible.