p-values and Significance
The most misused number in science
The p-value is the probability of data as extreme as observed under the null — small means inconsistent, not 'important'. Significance thresholds, multiple testing and effect size complete the picture.
What you will learn
- Define the p-value precisely
- Explain the multiple-testing inflation problem
- Interpret p alongside effect size, not alone
What is this?
The p-value is P(statistic ≥ observed | H₀ true) — the probability of seeing data this extreme if the null were really true. A small p-value says 'this data is surprising under H₀', which is evidence against H₀. It does NOT say the effect is real, large or important. With enough tests, small p-values appear by chance: this is the multiple-comparisons trap.
Why do you need it?
- People quote p=0.00001 as 'proof' when it's only consistency evidence.
- Running 20 hypotheses at α=0.05 will produce ~1 false positive on average.
- Correcting (Bonferroni, FDR) is mandatory for any dashboard with many metrics.
- Senior ML interviews test whether you know p is about data given null, not null given data.
Mental model
A coin landing heads 9 of 10 times
A small p-value
Under a fair-coin null, 9/10 heads is unlikely — but it doesn't make the coin 'very biased to 90%'. It primes suspicion and invites more experiments; it doesn't nail down the truth.
How it works
- Fix H₀ and a test statistic; the null distribution describes its behavior.
- Observed statistic → p = tail probability, e.g., P(t ≥ t_obs).
- Reject H₀ if p < α. α bounds long-run Type I error when H₀ is true.
- Multiple tests inflate the chance of at least one false positive: familywise error ≈ 1 − (1−α)ᵏ.
- Bonferroni divides α by k (conservative); FDR (Benjamini-Hochberg) controls the expected false-positive proportion.
Why 20 tests at α=0.05 find false positives.
In production
- A metrics dashboard corrects for multiple testing across 40 KPIs so a 'significant' weekly move is trustworthy.
- A genomics-scale ML platform uses Benjamini-Hochberg FDR on thousands of feature associations.
- A product team pairs p-values with practical effect sizes (lift %) before launching.
Best practices
- Report the effect size and CI alongside every p-value.
- Pre-register the metric(s) — the primary one is the one you test formally.
- Apply multiple-testing corrections whenever you scan many hypotheses.
- Treat p-values near 0.05 as 'inconclusive' rather than 'victory'.
Common mistakes
- Interpreting p as P(H₀ is true | data).
- Shoppings thousands of features with out corrections.
- Switching hypotheses after seeing results (garden of forking paths).
- Equating statistical significance with business significance.
Debugging tips
- If every experiment returns p < 0.05, you're either testing tiny effects with huge n (p doesn't mean important) or committing selection bias.
- Recompute the p-value vs a permutation null when assumptions might be violated.
- For dashboards of many metrics, sort by FDR-adjusted q-values, not raw p.
Interview checkpoint
Quick quiz
A p-value is the probability that:
Testing 30 dashboard metrics at α=0.05 needs a multiple-testing correction to trust the 'significant' ones.