A/B Testing
Intermediate 14 min read A/B Testing

Confidence Intervals for Experiments

Plausible effect sizes, not just 'is it significant'

A confidence interval shows the range of plausible true effects. Unlike a p-value, it communicates magnitude and uncertainty together — the practical tool for deciding whether an observed lift is big enough to matter.

confidence intervalmargin of erroreffect sizeliftstandard error
Prerequisites: P-values & Significance
Before we start

What you will learn

  • Compute a 95% confidence interval for a conversion lift
  • Interpret a CI without the '99% chance it contains the truth' reading
  • Use the CI to talk about practical significance and risk
In 60 seconds

A 95% confidence interval for the true lift is an interval computed from data such that, over repeated runs of the procedure, 95% of intervals will capture the true effect. The bracket around the point estimate communicates plausible magnitudes.

Introduction

What is this?

A confidence interval for a treatment effect is the range of true-effect values consistent with your data at a chosen confidence level. For a conversion lift, CI = estimate ± z_{α/2} × standard error. The interval tells you the direction (both bounds positive?), the magnitude (upper bound 13% vs 2%? it's a different business), and the precision (wide = uncertain, narrow = refined).

Motivation

Why do you need it?

  • A p-value says 'signals is weird under the null' but the CI says 'the effect is likely between +1% and +13%' — which the business can actually use.
  • An interval that hugs zero (e.g. +0.1% to +3%) says 'either tiny or nothing' — a different decision than a decisively positive bracket.
  • Reporting CIs is the antidote to star-chasing: it forces magnitude and uncertainty into the same sentence.
  • Wide CIs from underpowered tests become visible, making sample-size laziness obvious.
Think about it

Mental model

Imagine

A weather forecast range

Think of

A confidence interval

'High today 18–24°C' is honest about uncertainty the same way '+8% lift, 95% CI +3% to +13%' is. A single point '21°C' invites false precision.

Mechanics

How it works

  • Compute point estimates p̂_c and p̂_t and the difference d = p̂_t − p̂_c.
  • Compute the standard error for the difference of two proportions: se = √(p̂_c(1−p̂_c)/n_c + p̂_t(1−p̂_t)/n_t).
  • Build the interval: d ± 1.96·se (or t-critical for small samples).
  • Interpret: for a 95% CI, the procedure captures the true effect in 95% of repeats — not a probability about this particular interval.
  • Translate to relative lift: divide the difference CI by p̂_c to express as a percent lift.

From point estimate to interval.

Estimate
Standard error
Margin
Interval
Read
Hands-on

Examples

CI for the conversion lift

CI for the conversion lift
import math
c, nc, t, nt = 3000, 30000, 3250, 30000
pc, pt = c/nc, t/nt
d = pt - pc
se = math.sqrt(pc*(1-pc)/nc + pt*(1-pt)/nt)
lo, hi = d - 1.96*se, d + 1.96*se
print(f"absolute diff: {d*100:.2f}pp  (CI {lo*100:.2f} to {hi*100:.2f}pp)")
print(f"relative lift: {(pt/pc-1)*100:.1f}%  (CI {(lo/pc)*100:.1f}% to {(hi/pc)*100:.1f}%)")
# → absolute +0.83pp (CI +0.38 to +1.29); relative +8.3% (CI +3.8% to +12.9%)

The interval converts 'significant' into a usable statement: we're fairly confident the true lift is between roughly +4% and +13% — a market-sizeable range that also tells us the effect is not just noise.

Real world

In production

  • A pricing team decides to ship after seeing a CI wholly above their 3% revenue break-even threshold, not merely a significant p.
  • An experiment review board rejects a 'green' test because the CI spans −1% to +4%: plausible downside below zero, so launch is a bet not a read.
  • A platform uses CI width as a quality bar, refusing to report tests whose intervals are wider than the MDE.
Guidelines

Best practices

  • Report absolute and relative intervals together — teams live in percentages, costs live in points.
  • Prefer 95% at review but widen to 90% for the working discussion; never shop for the level that fits the story.
  • For skewed metrics (revenue, latency), bootstrap the CI instead of the normal approximation.
  • State the interval's width explicitly when it's wide — 'we know the effect is between +1 and +13%', i.e. rough.
Watch out

Common mistakes

  • Claiming 'there's a 95% chance the true effect is in this interval' — the interval is fixed after sampling; it's the procedure that has the 95%.
  • Building the CI on the pooled SE instead of the unpooled SE of the difference.
  • Reporting a CI in relative terms while decisions need absolute risk (or vice versa).
  • Dropping the CI from the report once it becomes inconveniently wide.
When things break

Debugging tips

  • A CI narrower than the point estimate's plausible spread usually means sample count was inflated by duplicate events.
  • If your CI and an online calculator disagree, compare whether you used pooled vs unpooled standard error.
  • Wide CI after full duration = high variance metric; consider a variance-reducing metric (e.g. CUPED) on the next test.
  • Check the CI for the guardrail metrics too — a green primary with a wide red-on-revenue interval is not green.
Interviews

Interview checkpoint

Practice

Hands-on lab

Bootstrap a Confidence Interval for an Experiment
30 min · intermediate
In progress
Objectives
  • Compute the exact analytic CI for a conversion lift
  • Resample the data to build a bootstrap CI and compare
  • See how skew breaks the normal approximation

Bootstrap a Confidence Interval for an Experiment

30 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A 95% CI for a lift is −0.5% to +4.5%. The correct read is:

2

An interval that excludes zero at 95% is the same as p<0.05 for a two-sided test.

Go deeper

Official documentation

Confidence Intervals — Stat Trek
docs
Evan Miller — CI for two proportions
docs