LOOCV and Repeated K-Fold
Estimates that cost more compute but tell a fuller story
Leave-one-out CV trains on n−1 rows and validates one, giving a nearly unbiased but high-variance estimate. Repeated (shuffled) k-fold trades compute for a tighter, more stable estimate.
What you will learn
- Explain LOOCV's near-unbiased but high-variance nature
- Use repeated shuffled k-fold to stabilize fold means
- Know when these strategies are worth the compute
What is this?
LOOCV (leave-one-out) fits n models, each trained on every row except one, validating that single row — maximum data reuse, near-zero training-set bias, but high variance because training sets differ by one point. Repeated k-fold re-runs shuffled k-fold s times and averages everything, trading more training runs for a tighter, more trustworthy estimate. Both are 'more CV', chosen when stability beats a single pass.
Why do you need it?
- On small datasets, every row matters; LOOCV uses all but one for training each time.
- Repeated folds give confidence intervals around the metric instead of point estimates.
- A single lucky shuffle can mislead; repetition hammers out shuffling noise.
- For expensive-to-acquire data, using nearly-all rows for training is information-preserving.
Mental model
Polling one person at a time vs polling many times
LOOCV vs repeated k-fold
LOOCV interviews everyone once, extremely carefully (n models). Repeated k-fold re-polls with fresh random subsets s times — more interviews overall, tighter confidence in the trendline.
How it works
- LOOCV: for i in rows — train on X without row i, predict row i; average the n errors.
- Because each training set is n−1 of the same n rows, the estimates are highly correlated — variance is often high despite the bias being minimal.
- Repeated k-fold: for rep in 1..s — shuffle, run k-fold, collect mean; average across s×k fits.
- The distribution of rep-level means gives a confidence interval for the true performance.
- Compute cost: LOOCV = n fits; repeated k-fold = s×k fits.
LOOCV vs repeated k-fold cost and stability.
In production
- A tiny clinical dataset (~60 labeled cases) uses LOOCV because no validation row can be spared.
- Selection of a final model uses 5-shuffle repeated 10-fold to decide between two close candidates with real confidence.
- Reporting for compliance includes CV quantiles (p5/p50/p95) generated by repeated folding.
Best practices
- Prefer repeated k-fold over LOOCV for almost everything except very small n.
- Fix seeds per repetition so runs are reproducible and fold assignments auditable.
- Report median and interquartile range of fold means, not just average.
- Limit LOOCV/repeated to linear or cheap models; for neural nets use fewer folds.
Common mistakes
- LOOCV on huge datasets (n fits!) and wondering why compute exploded.
- Reading LOOCV's low bias as low variance — it is famously noisy for unstable models.
- Not shuffling between repetitions, so 'repeated' runs the exact same folds.
- Reporting repeated-k-fold mean without the spread across repetitions.
Common issues
LOOCV is far worse than 10-foldModel unstable to single-row changes (trees with strong leaf dependence)
Use repeated stratified k-fold for a stabler estimate, or prune/regularize the model.
Repeated CV gives a different answer every run despite seedsUnseeded inner randomness (model or splitter)
Seed and log the splitter and estimator; rerun with only the seeds changing.
Debugging tips
- If repeated folds disagree hugely across seeds, your model is unstable — investigate before trusting any number.
- Plot per-repetition means: a widening spread means the estimate is not converged.
- Compare LOOCV vs k-fold means; a big gap hints at instability, not a bug.
- Check that 'leave one out' applies to rows — for groups use leave-one-group-out instead.
Interview checkpoint
Quick quiz
LOOCV fits how many models for an n-row dataset?
LOOCV has the lowest variance of all CV strategies.