What Is Cross Validation?

• Explain why a single split under-reports uncertainty
• Describe the re-fit-and-average loop of cross validation
• Know when one split suffices and when only CV will do

• One random split can be flattering or brutal depending on which rows landed in test.
• Model choice, hyperparameter tuning and feature decisions all need a reliable performance estimate.
• CV gives you the variance of the estimate, not just the mean — is 0.85 really 0.84-or-0.88?
• It's the honest currency used to compare between models before you commit.

One spoonful (a single split) can be unrepresentative — the pot isn't fully mixed. Sampling repeatedly across the pot (cross validation) gives you a much truer picture of the soup's quality.

• Split the data into k roughly equal folds (default k=5 or 10).
• Iterate k times: hold out fold i as validation, train on the other k−1 folds.
• Evaluate the trained model on fold i and record the metric.
• Average the k metrics → the CV estimate; the standard deviation of folds → estimate variance.
• Every record is used for validation exactly once and for training k−1 times.

• A team picks between two sklearn models using 5-fold CV mean AUC, and the 0.02 gap is larger than the fold spread — a defensible choice.
• A supplier of ML models is contractually required to report CV performance with quantiles, not a single number.
• Small-data teams (few thousand rows) lean on CV because a single 80/20 split wastes scarce validation rows.

• Keep k between 5 and 10: fewer folds, more bias per fold; more folds cost compute.
• Hold out a truly untouched test set for final confirmation — CV can still leak through repeated evaluation.
• Use stratified CV for classification so each fold keeps the class ratio.
• Report mean ± std of the folds, and note whether the split respects structure (groups, time).

• Cross-validating once, then 'doing CV again' on the winner — repeated peeking at the data inflates estimates.
• Shuffling time-series data randomly when the future must not train on the past.
• Forgetting that CV on a tiny test set is still not a guarantee of production performance (different distribution).
• Averaging folds that used different preprocessing or different training data as if equal.

• Per-fold metrics swinging widely? Check for group structure or rare-class imbalance that shuffling breaks.
• If one fold is always far worse, find which rows live in it — likely a rare subgroup.
• Compare CV mean to the single-split estimate: a big gap means your split was unrepresentative.
• Log fold seeds and fold assignments so any future metric debate is reproducible.