A/B Testing
Intermediate 15 min read A/B Testing

Sample Size and Power

The arithmetic that decides whether your experiment can work at all

Sample size is a function of baseline conversion, minimum detectable effect, significance level and power. Getting it right decides whether your experiment can ever see what you're testing for.

sample sizestatistical powerMDEalphabetatype II error
Prerequisites: Hypothesis & Metrics
Before we start

What you will learn

  • Explain power, alpha, beta and MDE in plain words
  • Compute a required sample size for a conversion experiment
  • Reason about why underpowered tests are worse than not testing
In 60 seconds

Sample size per group ≈ a constant × z-values × variance / MDE². Four inputs: baseline rate, the smallest effect you care about (MDE), the tolerated false-positive rate (alpha=5%) and the required power (typically 80%).

Introduction

What is this?

Sample size and power planning is the statistical preparation for an experiment. Power is the probability of detecting an effect when one exists (1 − β); alpha is the tolerated false-positive rate; the minimum detectable effect (MDE) is the smallest true improvement you want your test to reliably catch. The sample size needed is driven mostly by baseline variance and the MDE you demand — cut the MDE in half and you need 4x the sample.

Motivation

Why do you need it?

  • An underpowered test will show 'null' even when the change works — a type II error that silently kills good ideas.
  • Overpowered tests magnify tiny, commercially irrelevant differences into 'significant' launches.
  • Sample size planning tells you whether an idea is testable with your traffic at all.
  • Power analysis protects the whole pipeline: it converts vague 'give it two weeks' into an explicit promise.
Think about it

Mental model

Imagine

A leaky measurement cup

Think of

A low-power experiment

If your cup leaks a splash every measure (high variance), you need many more pours to be sure one cup actually has more water. Power analysis tells you how many pours.

Mechanics

How it works

  • Set alpha (false-positive tolerance, usually 0.05 two-sided) and power (1 − β, usually 0.80).
  • Get the baseline rate p0 (e.g. 10% conversion) and pick the MDE you care about (e.g. relative +10%).
  • Use the two-proportion sample size formula: n ≈ (z_{α/2}√(2p̄(1−p̄)) + z_β√(p0(1−p0)+p1(1−p1)))² / (p1−p0)².
  • Multiply by 2 for two groups; compute runtime = n / expected daily eligible traffic.
  • If runtime is too long, widen MDE, accept lower power, or reduce variance (more precise metric).

How MDE and variance control required sample size.

Baseline p0
MDE
alpha / beta
variance
n per group
Hands-on

Examples

Two-proportion sample size in Python

Two-proportion sample size in Python
import math
p0 = 0.10          # baseline 10%
mde_rel = 0.10     # +10% relative
p1 = p0 * (1 + mde_rel)
alpha, power = 0.05, 0.80
z_alpha = 1.96     # two-sided
z_beta  = 0.84
pbar = (p0 + p1) / 2
numer = z_alpha*math.sqrt(2*pbar*(1-pbar)) + z_beta*math.sqrt(p0*(1-p0)+p1*(1-p1))
denom = (p1 - p0)**2
n_per_group = numer**2 / denom
print(f"n per group ≈ {math.ceil(n_per_group):,}")
# → n per group ≈ 31,452  (with 10k/day → ~6.3 days plus ramp)

The formula balances the false-positive budget, the power you want, and the variance (encoded in p(1−p)) against the smallest effect you demand. Small MDE = huge n, which is why teams tune the MDE to what is commercially meaningful.

Real world

In production

  • A low-traffic B2B site discovers a checkout A/B needs 6 months of traffic to detect a 2% lift — it reframes the test as a 4-week directional pilot or waits for seasonal traffic.
  • A high-traffic mobile app detects a 0.1% lift in 2 days and runs rolling guardrail-style experiments continuously.
  • An e-commerce team splits duration math with ramp/learning phases, adding 20% buffer for novelty and holdout.
Guidelines

Best practices

  • Pre-register sample size and effect direction before seeing any data.
  • Choose the MDE by business value, not convenience: the smallest lift worth building for.
  • Compute per-user or per-unit at the randomization grain — don't mix granularities.
  • Re-derive sample size when a metric's variance changes (e.g. after platform migration).
Watch out

Common mistakes

  • Using 95% power 'to be safe', making tests perennially underpowered by practical constraints.
  • Computing sample size per session when metrics are per user (or vice versa).
  • Ignoring the difference between relative and absolute MDE in team discussions.
  • Starting the test and 'seeing how it goes' — post-hoc 'Moore's law of sample size' always produces garbage.
When things break

Debugging tips

  • If a test hits its sample size but p is still around 0.5, check for a weak MDE or high metric variance.
  • Verify baseline p0 from the last 30 days, not a branded promo period.
  • If the experiment tool disagrees with your number, check its variance and its assumed MDE definition.
  • When traffic is insufficient, test on a higher-volume segment instead of stretching the window.
Interviews

Interview checkpoint

Practice

Hands-on lab

Compute Sample Size and Check Your Test's Power
30 min · intermediate
In progress
Objectives
  • Compute required sample size from baseline, MDE, alpha and power
  • Simulate an experiment and measure empirical power
  • Observe how MDE and variance drive sample size

Compute Sample Size and Check Your Test's Power

30 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Halving the minimum detectable effect (MDE) roughly multiplies the required sample size by:

2

Power is the probability of detecting an effect when one truly exists.

Go deeper

Official documentation

Evan Miller — sample size calculator
docs
Sample Size Determination — NIST handbook
docs