Statistics
Advanced 14 min read Statistics

Statistical Power and Effect Size

Can this experiment possibly detect the effect?

Power is the chance a test detects an effect that exists. Effect size says how big the effect is. Together they make the sample size: what you'll be able to conclude before spending an experiment.

powereffect sizesample sizepower analysisType IIMDE
Prerequisites: Hypothesis Testing Sampling and Inference
Before we start

What you will learn

  • Compute power and effect size
  • Run a sample-size calculation for a test
  • Interpret 'not significant' with power in mind
Introduction

What is this?

Effect size is a standardized magnitude of the effect (e.g., Cohen's d = (μ1−μ2)/σ). Power is the probability that the test rejects H0 when the effect is real, given sample size and α. Nailing power before running an experiment sets the minimum detectable effect (MDE) you can honestly detect — and tells you when a null result is real evidence versus an underpowered shrug.

Motivation

Why do you need it?

  • Most 'no difference' results are underpowered noise, not truth.
  • Power analysis prevents wasting engineering on doomed experiments.
  • Effect size interpretation rescues results from the noise of significance alone.
  • MDE framing communicates risk to stakeholders: what we can and cannot detect.
Think about it

Mental model

Imagine

A telescope's resolving power

Think of

Statistical power

A small telescope can't see faint moons (small effects) from far (small samples). You choose the telescope by the smallest moon you must detect — that's choosing sample size from MDE.

Mechanics

How it works

  • Choose α (risk of false positive), desired power (usually 80–90%), and the effect size of interest.
  • Estimate baseline variability (σ) from historical data.
  • Solve the sample-size formula: n ≈ 2·(z_{1-α/2}+z_{power})²·σ² / δ².
  • Smaller detectable effect δ → vastly bigger n (δ in the denominator, squared).
  • After the test, compute achieved power: how conclusive a null really was.

The power tradeoff.

Choose α + power
Effect size δ
Baseline σ
Sample size n
Run test
Hands-on

Examples

A sample-size calculation

A sample-size calculation
import math
alpha, power = 0.05, 0.8
z_a = 1.96; z_b = 0.84
sigma = 12.0          # historical SD of the metric
delta = 2.0           # smallest %-change you care about detecting
n = 2 * (z_a + z_b)**2 * sigma**2 / delta**2
print("n per group ≈", math.ceil(n))   # ≈ 283
 
# Double the needed sensitivity  (halve delta):
delta2 = 1.0
print("n for half the effect ≈", math.ceil(2 * (z_a+z_b)**2 * sigma**2 / delta2**2))

The formula makes the brutal math visible: needing to detect half the effect quadruples the sample size.

Real world

In production

  • An A/B pipeline blocks launch until the sample size for the MDE is hit.
  • A guardrail review uses power to decide if a 'no regression' result can be trusted.
  • Forecast teams report 'we can detect ≥5% swings; below that is noise'.
Guidelines

Best practices

  • Compute required n BEFORE the experiment ships.
  • Set MDE from business stakes, not from guesswork; ask 'what lift matters?'
  • For a null conclusion, report achieved power; underpowered nulls are worthless.
  • Use one-sided power thinking for guardrails (fail-only direction).
Watch out

Common mistakes

  • Shipping tests too small to ever detect the promised lift.
  • Reading an underpowered null as proof that there is no effect.
  • Choosing δ after seeing results to justify the n you used.
  • Ignoring variance inflation from metric we have heavy users.
When things break

Debugging tips

  • 'We saw no lift' with n tiny — compute power to frame the statement correctly.
  • Variance far from expectation after launch — revisit σ; the plan was wrong, not the effect.
  • Desired n astronomically large — the effect you asked to detect is too small; revisit the ask.
  • Use winsorization/robust metrics when outliers inflate σ beyond the plan.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

To detect HALF the effect size with the same power, the sample size must roughly:

2

A 'no significant difference' result from an underpowered test proves the treatments are equal.

Go deeper

Official documentation

R: pwr package examples
docs
statsmodels power module
docs