Multiple Comparisons and Segmentation Bias
Why slicing an experiment into 50 segments invents 2.5 winners
Test enough segments and some will be 'significant' by chance. Multiple-comparisons control — Bonferroni, Holm, FDR — and disciplined pre-registration keep that fact from manufacturing fake insights.
What you will learn
- Explain the multiple-comparisons problem in one sentence
- Apply Bonferroni, Holm and Benjamini–Hochberg corrections
- Recognize post-hoc segment mining as the pipeline's main fake-insight generator
Each test has a 5% false-positive chance. Run 20 segment tests and you expect about one 'significant' result even when nothing is real. Corrections or pre-registration are required before a segment 'win' means anything.
What is this?
The multiple-comparisons problem is the inflation of false positives when many tests are run on overlapping data: at α=0.05, m independent tests yield roughly m×0.05 false discoveries. Segmentation bias is its most common form — slicing an experiment by device, plan, campaign or segment until something looks green. Control methods (Bonferroni, Holm, Benjamini–Hochberg FDR) either tighten the per-comparison threshold to hold the family-wide error, or explicitly bound the expected share of false discoveries among the 'wins'.
Why do you need it?
- A 'new customers on Android' win found by scanning 40 segments is indistinguishable from luck.
- Companies burn budgets on segment-targeted rollouts that were pure noise.
- FDR methods give you more power than Bonferroni while remaining principled — good analysis needs them.
- Pre-registering segment hypotheses converts mining into testing, which is the only honest defense.
Mental model
Twenty lie detectors
Twenty tests on one experiment
Each detector has a false-alarm rate. Give twenty of them the same innocent person and expect an alarm or two by pure mechanics — that alarm is a multiple-comparison false positive.
How it works
- Enumerate how many tests you're really running (segments × metrics × stops).
- Choose a control strategy: strong family control (Bonferroni: α/m; Holm: sequential) or FDR (Benjamini–Hochberg: rank p-values and compare p_j ≤ (j/m)α).
- Pre-register the segments and hypotheses that count; anything discovered post-hoc is labeled discovery, not evidence.
- Sensitive segments: adjust winner claims for segment size (a 'win' in a 2% subset may be economically irrelevant).
- Report corrected thresholds with the numbers so the review room sees the math, not just stars.
Multiple tests inflate false positives until corrected.
Examples
Benjamini–Hochberg in five lines
import numpy as nppvals = np.array([0.001, 0.02, 0.07, 0.20, 0.80]) # five segment testsm = len(pvals)order = np.argsort(pvals)sorted_p = pvals[order]alpha = 0.05# flag significant where p_j <= (j/m)*alphasig = sorted_p <= (np.arange(1, m+1)/m)*alphafor p, j, s in zip(sorted_p, np.arange(1, m+1), sig): print(f"p={p:.3f} bound={(j/m)*alpha:.3f} -> {'FDR significant' if s else 'n.s.'}")# p=0.001 and p=0.02 survive; 0.07 does notB–H controls the expected proportion of false positives among the rejected tests (FDR). It rejects progressively until the largest p that meets the (j/m)α line, keeping far more power than Bonferroni while staying honest about the 5 segment claims.
In production
- An e-commerce team pre-registers 6 segments (device, new/returning, country, plan) before the test; the 30 post-hoc slices they were tempted by get flagged as hypothesis-generating only.
- A growth team explores 24 channel-segment effects with Benjamini–Hochberg and ships only the two that survive 10% FDR, then validates with a follow-up test.
- An experimentation platform auto-applies Bonferroni when dashboards display more than 10 side-by-side metric comparisons.
Best practices
- Pre-register segments: which slices you'll trust, before you see them.
- Adjust when you decide to report many segments; the number of tests is a design decision, not a constant.
- Prefer FDR (BH) for exploratory segment reporting and Bonferroni/Holm for a small set of confirmatory hypotheses.
- Always size the segment: a corrective p in 3% of users may be unactionable.
Common mistakes
- Scanning 40 segments and celebrating the one with p<0.05 — that's the expected harvest of noise.
- Applying Bonferroni to 200 tests and concluding 'nothing works', destroying power for real signals.
- Correcting 15 metrics but calling the 16th 'the guardrail' and not adjusting — the guardrail is a comparison too.
- Using 'but I only thought of the segment after seeing the data' as evidence of insight rather than mining.
Debugging tips
- Count every test you ran, including reruns of the same segment with different filters — those count as tests.
- If a segment 'win' is about 5% of users, re-check that the segment definition wasn't chosen after scan.
- Reproduce segment p-values from the raw events; BI-vendor segment tables often apply different dedupes.
- For exploratory claims, require a confirmatory follow-up before budget is spent.
Interview checkpoint
Hands-on lab
- Show raw segment mining producing 'fake wins'
- Apply Bonferroni and Benjamini–Hochberg to the same p-values
- See how correction changes which segments you'd trust
Correct Multiple Comparisons on a Simulated Experiment
30 min · advanced
Quick quiz
A team runs 20 segment tests at α=0.05 with no correction. Expected number of false-positive 'wins'?
Skipping a multiple-comparisons correction and instead 'only looking at the significant ones' is a valid approach.