Cross Validation
Intermediate 14 min read Cross Validation

Stratified and Group Splits

Splits that respect class balance and entity structure

When classes are imbalanced or rows share a group, plain k-fold misrepresents the data. StratifiedKFold preserves class ratios; GroupKFold keeps every member of an entity on one side of the split.

StratifiedKFoldGroupKFoldgroup leakagestratificationmultilabel
Prerequisites: K-Fold Cross Validation
Before we start

What you will learn

  • Choose stratified splitting when classes are imbalanced
  • Explain why repeated entities need group splits
  • Spot group leakage before it waters down your estimate
Introduction

What is this?

Plain k-fold ignores two truths about real data: labels are imbalanced and rows are not independent. StratifiedKFold shuffles then partitions while preserving the target distribution in every fold, so rare classes are always represented in validation. GroupKFold partitions by a group variable instead, guaranteeing that all rows of a user, patient or session stay together in either train or test — eliminating group leakage where the model 'recognizes' entities it saw during training.

Motivation

Why do you need it?

  • Unstratified folds can drop a rare class entirely from validation, making fold metrics meaningless.
  • Correlated rows inflate realistic difficulty: a split across the same user's sessions is deceptively easy.
  • Fraud, recommendations, medical and multi-image tasks all have natural groups.
  • Group-stratified CV yields the honest estimate of what will happen to new entities.
Think about it

Mental model

Imagine

Cooking for a family with allergies

Think of

Stratified + group splitting

You can't just split the meal any way — each served portion must include safe options for everyone (stratify on the dietary label) and never switch a guest's dish mid-meal (keep a person's courses together).

Mechanics

How it works

  • StratifiedKFold partitions shuffled indices so each fold's label histogram mirrors the full dataset.
  • For multilabel targets it stratifies on the multilabelization so each combination appears in every fold.
  • GroupKFold gathers all indices sharing a group_id and assigns whole groups to folds — groups never split.
  • The number of folds is bounded by the number of groups, not rows.
  • Combining both (class ratio AND group integrity) is done by GroupShuffleSplit-style or custom splitters.

Rows vs groups as the unit of the split.

KFold
StratifiedKFold
GroupKFold
Group-stratified
Real world

In production

  • A fraud model splits by cardholder so the same person's transactions never cross the fold boundary.
  • An image classifier splits by subject ID, not image, so a person's photos don't leak into both train and test.
  • A churn model stratifies folds by churn label because churners are only 4% of accounts.
Guidelines

Best practices

  • Prefer StratifiedKFold for single-label classification out of the box.
  • Add group splits whenever you can name the entity that generates multiple rows.
  • Log the split strategy in the experiment — 'KFold' vs 'GroupKFold' changes meaning.
  • When groups and labels interact (rare amid common), use a stratify-within-group scheme.
Watch out

Common mistakes

  • Using KFold on highly imbalanced labels and trusting fold means.
  • Grouping by row index instead of the true entity id, so groups are meaningless.
  • Splitting by group but forgetting that stratification — both desirable — requires a combined splitter.
  • Building folds from groups but training row-wise features that already encode group identity.
Root cause analysis

Common issues

Error
Model says 0.99 on CV, fails on new customers
Root cause

Row-correlated instances straddled folds, letting the model memorize identities

Fix

Switch to GroupKFold keyed on the true entity id and drop entity-id features that leak membership.

Error
GroupKFold errors when groups outnumber samples
Root cause

Each group has one row; the splitter can't partition

Fix

Verify group uniqueness; if every group is unique, the data may have no group structure and KFold is fine.

When things break

Debugging tips

  • Print per-fold class counts; any fold at 0 on a class is an emergency, not a surprise.
  • Check for group id collision: same entity id with different feature shapes signals dirty keys.
  • If CV estimates look too good, ask 'do these rows share an entity that a new user won't share?'
  • Verify fold size variance — GroupKFold folds can differ a lot in row count if groups are skewed.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Your data has 10 transactions per user. To evaluate on new users, split with:

2

StratifiedKFold fixes group leakage.

Go deeper

Official documentation

sklearn cross-validation splitters
docs