Statistical Tests and Explainability
Signals from noise, proved out loud
Statistical tests separate signal from noise — compare groups, banners, cohorts honestly — while explainability keeps every model's output provable and plain-spoken. Together they keep BI honest.
What you will learn
- Run a statistical comparison with honest interpretation
- Interpret model output with feature importance, not vibes
- Detect and report bias in scores and segments
What is this?
Statistical tests give the 'is this real?' verdict: comparing two groups, banners, or cohorts with proper assumptions and honest p-values — not 'felt different'. Explainability makes model output provable: feature importance, band definitions, plain-language 'why' notes per score, and bias scrutiny. Together they keep BI's core promise — a number you can hold accountable to a method.
Why do you need it?
- Cohort growth can be an illusion; a proper test says so before the meeting.
- Exec starts asking 'why this score' the moment a model contradicts intuition.
- Bias detection is a governance duty, not a nicety.
- Stats + explainability is the credibility layer of whole BI teams.
Mental model
A referee's whistle and a body-cam
Tests and explainability
The referee's whistle says whether the play was a foul (the statistical test) — the body-cam explains the call (the explainability). Neither alone is enough: a whistle without video feels arbitrary; video without whistle analyzes nothing. BI needs both: verdict and the reasoning behind it, replayable.
How it works
- Form the hypothesis: 'region A conversion > B' with a null and a guardrail.
- Check assumptions: independence, sample size, distributions — before the p-value.
- Test: appropriate test chosen by data shape (t-test-like, proportion, or a chi-square-style check).
- Interpret honestly: p-value is evidence, not a Rorschach; report effect size too.
- Explaining models: feature importance + per-score reasoning + bias audit, shipped with output.
A test result and its explainability contract.
Examples
Comparing banners honestly
TEST new banner vs control conversion control 4.1% (n=10k), new 4.6% (n=10k) z-proportion test: p=0.08, lift +0.5ptVERDICT not significant at 0.05 — 'different' is unproven; the +0.5pt is within fatigue. effect size small; run another week before the meeting quotes it.EXPLAIN if shipped: banner D picked because it had the best holdout lift; drivers: hero text, CTA position; bias audit: no segment skew.Assumptions checked, test matched, verdict honest — effect size and bias noted — a verdict you can replay.
In production
- Banner/A/A/B ship decisions gated on significance + effect size, not the fanciest chart.
- A churn-score writeup lists top-5 features per band so the call center can argue productively.
- A bias audit flags segment drift in a segmentation every quarter.
Best practices
- State the null and check assumptions before touching a p-value.
- Report effect size and confidence with every verdict.
- Ship feature importance + per-score reasoning with every model output.
- Audit for bias by segment; politics-proof the scores.
Common mistakes
- p-value window-shopping — hunting until <0.05, then 'it works!'.
- Ignored assumptions (independence, sample size) making the test decorative.
- Scores without explanation — a black box demanding blind trust.
- Bias unchecked — a model quietly redistributing who gets flagged.
Debugging tips
- 'Significant' result that flips next run → sample size or peeking (multiple looks); fix the design.
- Team disputes a score → missing explain line; add drivers + band + confidence.
- Segment skew in scores → bias audit due; compare distribution per segment.
Interview checkpoint
Quick quiz
Before running a statistical test you should:
A model score without a feature-importance note is acceptable in BI.