A/B Testing
Intermediate 14 min read A/B Testing

P-values and Significance

The most quoted — and most misread — number in analytics

A p-value is the probability of seeing a result this extreme if there were truly no effect. It is not the probability the variant is better — a distinction A/B teams routinely botch.

p-valuesignificancenull hypothesisfalse positivestatistical significance
Prerequisites: Experiment Design
Before we start

What you will learn

  • State exactly what a p-value does and doesn't mean
  • Run a z-test and interpret the p-value correctly
  • Explain why 'not significant' is not 'no effect'
In 60 seconds

p = P(observed result or more extreme | no true effect). Small p means the data is surprising under the null, which pushes you toward believing a real effect — it never directly gives the probability the effect is real.

Introduction

What is this?

A p-value is the probability of observing your result (or something more extreme) assuming the null hypothesis is true — i.e. assuming there is genuinely no effect. 'p < 0.05' is a decision threshold, not a measure of effect size or practical importance. Significance is purely a statement about how surprising the data is under 'no effect', which is why 'statistically significant' and 'practically significant' are two different questions.

Motivation

Why do you need it?

  • The 'p = probability my variant is better' misreading is the most common — and most expensive — analytics error.
  • Decision thresholds turn ambiguous evidence into published decisions, but the decision needs effect size + context.
  • Conflating significance with importance sends teams chasing 0.1% 'wins' and passing 8% real wins as 'no effect'.
  • Reading p-values correctly is the difference between a disciplined experimentation culture and a superstitious one.
Think about it

Mental model

Imagine

A fingerprint comparison

Think of

A p-value

If a detective matching a print assumes two people are the same person (null) and still finds an impossible match rate, the probability of that under the null is tiny — a small p-value. It supports 'not the same person', but the detective still has to decide how important the match is.

Mechanics

How it works

  • State H0: no difference in true conversion between groups.
  • Compute the standard error from the pooled proportion and the z-statistic = (p̂_t − p̂_c)/SE.
  • Map the z-statistic to a two-sided tail probability — that's the p-value.
  • Compare to α = 0.05; 'significant' means the result is surprising-to-unlikely under no effect.
  • Report the p-value together with the effect size and confidence interval, never alone.

Where a p-value lives in the z-test pipeline.

Compute metric
Standard error
z-statistic
Tail probability
Compare α
Hands-on

Examples

Two-proportion z-test by hand

Two-proportion z-test by hand
import math
c = 3000; n_c = 30000   # control: 5,000 conversions wait, 3,000 / 30,000 = 10%
t = 3250; n_t = 30000   # treatment
 
pc, pt = c/n_c, t/n_t
p_pool = (c + t)/(n_c + n_t)
se = math.sqrt(p_pool*(1-p_pool)*(1/n_c + 1/n_t))
z = (pt - pc)/se
# two-sided p-value via the normal cdf
from math import erfc
p_value = erfc(abs(z)/math.sqrt(2))
print(f"conv: {pc:.3f} vs {pt:.3f} | z={z:.2f} | p={p_value:.4f}")
# → conv: 0.100 vs 0.108 | z≈3.55 | p≈0.0004 → significant

The z-score is the signal (delta in conversion) divided by the noise (standard error). p≈0.0004 says: if there were truly no effect, results this extreme would appear 4 times in 10,000 — strong evidence against 'no effect'.

Real world

In production

  • An experimentation team reports lift with a bracket: '+3.1% (95% CI +1.2 to +5.0, p=0.002)' — effect and evidence, not just a star.
  • A product team rejects a 'statistically significant' 0.05% engagement lift because the MDE they cared about was 2%: significant, but commercially irrelevant.
  • A company recognizes a promising-null pattern and runs a simple Bayesian re-analysis to quantify evidence for the effect.
Guidelines

Best practices

  • Always report effect size and CI next to the p-value.
  • Use the pre-registered alpha and don't re-interpret it after seeing data.
  • Treat 'p>0.05' as insufficient evidence, not proof of no effect — that requires an equivalence or power analysis.
  • Distinguish statistical from practical significance explicitly in the decision summary.
Watch out

Common mistakes

  • 'p=0.03 means a 97% chance the treatment is better.' — wrong; that's the posterior probability and needs a prior.
  • 'p=0.06 means there's probably no effect.' — wrong; evidence is inconclusive, direction may still be informative.
  • Comparing p-values across experiments to rank which won more.
  • Using one-sided tests as a backdoor to 'more significant' results without pre-registration.
When things break

Debugging tips

  • A suspiciously tiny p-value (p<0.0001) in a low-traffic test usually means wrong variance — check you used pooled proportions, not standard deviations of the data.
  • When two tools disagree on p, compare their SE formula and whether they used a continuity correction.
  • If significance flips when you change the sample filter, your population definition is unstable — fix the query, not the p.
  • p=0.049 vs p=0.051 should not change decisions; if it does, your decision rule is inhumane — widen the reporting bracket.
Interviews

Interview checkpoint

Practice

Hands-on lab

Run a Z-Test and Swap Misreadings for Correct Ones
30 min · intermediate
In progress
Objectives
  • Compute conversion lift, standard error, z and p-value in Python
  • Show what p-values mean under simulated null worlds
  • Observe that 'not significant' happens even with real effects when power is low

Run a Z-Test and Swap Misreadings for Correct Ones

30 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

p=0.02 means:

2

A result that is statistically significant is always practically important.

Go deeper

Official documentation

ASA statement on p-values
docs
p-value — Stat Trek
docs