Cross Validation
Advanced 16 min read Cross Validation

Nested Cross Validation

Measuring the tuning procedure, not a tuned run

Tuning inside plain CV leaks selection optimism into your estimate. Nested CV keeps model selection inside an inner loop and outer folds measure the honest performance of the whole tuning procedure.

nested cross validationmodel selectionoverfittingselection biastuning
Prerequisites: K-Fold Cross Validation Hyperparameter Optimization
Before we start

What you will learn

  • Explain the optimism that regular CV adds when you tune on it
  • Build inner/outer nested CV correctly
  • Recognize when nested CV is mandatory
Introduction

What is this?

Ordinary CV becomes optimistic the moment you use its folds to pick hyperparameters: the chosen settings are selected to score well on those very folds, so the reported number understates true error. Nested CV fixes this by splitting evaluation into two rings — an outer loop reports performance, and inside each outer fold an inner CV selects hyperparameters. The outer score estimates what this whole 'tune then release' procedure would earn on new data.

Motivation

Why do you need it?

  • Any tuning process that looks at validation folds contaminates them.
  • Teams routinely ship models whose reported metrics were measured on guiding-the-tuning data.
  • Nested CV is the estimate you can show an auditor: it measures the pipeline, not a lucky run.
  • It exposes whether tuning actually helps or is just curve-fitting noise.
Think about it

Mental model

Imagine

A coach using the same scrimmages to both choose starters and predict the season

Think of

Tuning and validating on the same folds

You game better against opponents you've practiced with. Nested CV is like picking starters on practice film (inner) but scoring the season against fresh opponents (outer).

Mechanics

How it works

  • Outer loop: split data into O folds; each outer fold is a quarantine set for reporting.
  • For each outer training portion, run an inner CV (e.g., 5-fold) that tunes hyperparameters and picks the best config.
  • Retrain the best-config model on the whole outer training portion and evaluate on the outer validation fold.
  • Average outer fold scores → honest estimate of 'tune-then-deploy' generalization.
  • The best final config is chosen by the inner search on ALL data (its score is optimistic — outer results are the truth).

Inner rings tune, outer rings report.

Outer fold
Inner CV
Best config
Evaluate
Report
Real world

In production

  • A model-selection report for a regulated product shows the outer-fold distribution, not the inner search score.
  • A platform runs nested CV nightly so auto-tuning doesn't silently inflate published model metrics.
  • A team decides 'does bayesian tuning beat default?' by comparing nested-CV scores of both procedures.
Guidelines

Best practices

  • Match inner and outer CV structure (stratified, grouped) to the data's constraints.
  • Report outer-fold mean with spread and the per-fold best configs.
  • Use nested CV at least once before finalizing any hyperparameter-tuned system.
  • Document that the shipped model's true expected performance equals the outer score, not the search score.
Watch out

Common mistakes

  • Running nested CV but reporting the inner search score anyway.
  • Stopping tuning when the outer score stops improving but keeping search score in slides.
  • Using different split strategies inside/outside, so the layers don't represent the same procedure.
  • Skipping nested CV 'for speed' on expensive models, then inheriting optimistic metrics for months.
Root cause analysis

Common issues

Error
Nested CV outer score is much lower than the tuning report
Root cause

That's expected — tuning score is optimistic; the outer score is the honest estimate

Fix

Report the outer estimate for decisions; optionally invest in a better inner search to raise honestly-perceivable performance.

Error
Outer folds keep disagreeing on best hyperparameters
Root cause

Search space too wide, data too small, or tuning is noise-driven

Fix

Narrow the space, add domain priors, and verify on a large inner budget before trusting the winner.

When things break

Debugging tips

  • Compare plain-CV score vs outer nested score; a large gap confirms selection optimism was present.
  • If outer folds show different best configs, treat tuning as unstable and add search budget or shrink the space.
  • Log both loops' randomized states so any reported outer score is reproducible.
  • When inner search is slow, sample fewer configs first — nested CV amplifies the search budget's cost.
Interviews

Interview checkpoint

Practice

Hands-on lab

Quantify the Optimism Nested CV Removes
35 min · advanced
In progress
Objectives
  • Show plain CV's score inflates after tuning
  • Implement inner/outer nested CV
  • Compare the honest outer estimate against the optimistic tune score

Quantify the Optimism Nested CV Removes

35 min · advanced

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The outer score of nested CV measures:

2

Nested CV is only needed when hyperparameters are tuned automatically.

Go deeper

Official documentation

sklearn nested cross-validation example
docs