p-values and Significance
The most quoted, most abused number in analytics
p-values measure evidence against a null in a given model. Multiple comparisons, p-hacking and threshold BS (mangrove at 0.05) make them dangerous; the fix is pre-registration, correction, and reporting effect sizes and CIs.
What you will learn
- Interpret p-values within their assumptions and limits
- Avoid p-hacking and multiple-comparison traps
- Report significance responsibly with effect sizes and CIs
What is this?
A p-value is the probability of observing data as extreme as yours under a specific null model — no more. Significance at α=0.05 is a convention, not proof. The p-value inherits every assumption you made, is vulnerable to fishing (run 20 tests, one will pass), and says nothing about magnitude. Correct practice: pre-register, correct for multiple comparisons, and always present effect size and intervals.
Why do you need it?
- Managers cite p<0.05 as if it transfers belief; it only transfers surprise.
- Multiple comparisons turn 5% false positives into near-certain fireworks.
- p-hacking by peeking and trimming creates false certainties that ship bad features.
- Robust reporting (effects + CIs + pre-registration) is what senior stakeholders trust.
Mental model
A coin that's fair but occasionally lands heads 8/10
A p-value
p=0.055 says 'a fair coin would do this only 5.5% of the time'. It's a hint, not a verdict. Decide how surprising you need it to be (α) BEFORE flipping, or you're just rearranging the coin.
How it works
- p is computed conditional on H0 AND all model assumptions (independence, distribution).
- α is the pre-chosen rejection threshold that controls the Type I error rate long-run.
- Multiple testing inflates familywise error; Bonferroni/Holm or FDR control it.
- Effect size and CI carry the practical meaning the p-value cannot.
- Pre-registration commits to the plan, preventing result-dependent decisions.
Where p-values fit — and where they don't.
Examples
Multiple testing inflation
import numpy as np# 20 independent tests where H0 is TRUE everywhere:np.random.seed(7)p_vals = [np.random.uniform(0, 1) for _ in range(20)]signif = [p < 0.05 for p in p_vals]print("significant by chance:", sum(signif), "(expected ≈1 by 5% rule)") # exactly the wildfire riskRunning many tests guarantees false hits; correction tools (Bonferroni/FDR) are what tame the risk.
In production
- QA suites: dashboards flag 'significant anomaly' only after multiple-test correction.
- Experiment platforms apply FDR across many metrics before showing users a 'winner'.
- Research teams pre-register α and the primary metric before the test runs.
Best practices
- Pre-register the primary metric, test and α before collecting data.
- Correct p-values across all metrics and segments (FDR/Bonferroni).
- Report the effect size and its confidence interval with every p-value.
- For sparse or skewed metrics, prefer bootstrapped CIs over normal approximations.
Common mistakes
- Peeking every day and stopping the test when p crosses 0.05.
- Running 30 dashboard tests and celebrating 2 'wins'.
- Reporting p<0.05 as 'the effect is large' (it can be tiny).
- Tweaking exclusion rules after seeing results to flip the verdict.
Debugging tips
- 'So many significant results' — suspect un-corrected multiple testing.
- p stuck at exactly 0.049 across runs — suspicious; inspect model choices.
- Verdict flips when you add one metric — that's a fishing-broken process, not a finding.
- Document the full test plan in the notebook so reviewers can audit decisions.
Interview checkpoint
Hands-on lab
- Show that chance produces 'significant' results
- Run a multiple-testing simulation and count false positives
- Apply a Bonferroni correction and watch the count drop
Demonstrate p-Hacking and Multiple Testing
30 min · advanced
Quick quiz
Reporting 3 of 30 metrics as significant without correction is risky because:
p < 0.05 means the effect size is large.