A/B Testing Fundamentals
Randomize, isolate, compare
A/B testing compares two variants with randomized assignment and statistical inference. Its core move is controllability: only the variant differs, so any outcome difference can be attributed to it.
What you will learn
- Explain why randomization enables causal claims in A/B tests
- Describe the anatomy of an experiment: variants, metric, unit, duration
- Recognize where A/B tests apply and where they don't
An A/B test randomly splits eligible users into control and treatment, ships a change to treatment only, and compares the outcomes. Because assignment was random and nothing else differs, differences in outcome are attributed to the change.
What is this?
A/B testing (split testing) is controlled experimentation: a population is randomly divided into a control group (status quo) and one or more treatment groups (a candidate change). Each group's outcome metric is measured over a fixed period and compared with statistical methods to decide whether the observed difference is real or could be chance. Its power is causal: random assignment eliminates the systematic confounds that plague observational analysis.
Why do you need it?
- Observational analysis is full of confounders — 'users who used feature X retain better' is not 'feature X causes retention'.
- Randomization makes the groups exchangeable except for the variant, so differences are attributable.
- Experiments are reversible and cheap compared to full rollouts and give a decision rule with quantified risk.
- An experimentation culture builds a feedback loop that beats roadmap-by-opinion.
Mental model
A drug trial
An A/B test
Patients are randomized to drug or placebo, blinded where possible, and outcomes compared with p-values and power — product A/B testing is the same machinery applied to UI, pricing and copy.
How it works
- Enumerate eligible units (users, accounts, sessions) and randomize assignment (typically 50/50 or traffic-weighted).
- Ship the treatment to the treatment group; ideally neither the user nor the evaluator knows which (blinding).
- Collect the outcome metric over a pre-decided duration, at least the sample-size-derived minimum.
- Run a hypothesis test comparing the metric between groups and read a confidence interval.
- Decide: launch, kill or iterate based on statistical significance, effect size and guardrails.
Anatomy of an A/B test.
In production
- An e-commerce site A/B tests checkout layout; conversion lifts 3.2% (p<0.01) with no returns impact, and it ships.
- A SaaS runs a pricing experiment on a random 10% of trials, measuring activation and paid conversion before global rollout.
- A mobile network operator tests notification copy, watching uninstall rate as a guardrail against push annoyance.
Best practices
- Pre-register the hypothesis, primary metric and sample size before the test starts.
- Randomize at the level that matches the attribution (users for user metrics, sessions only for session metrics).
- Run the test the full planned duration; don't peek and stop early.
- Watch guardrail metrics (revenue, latency, churn, quality) alongside the primary metric.
Common mistakes
- Running the test until significance appears (peeking) — inflates the false-positive rate.
- Testing two things at once inside one 'A/B' and misattributing the effect.
- Randomizing at session level while measuring per-user outcomes.
- Ignoring novelty and primacy effects that fade or appear after the test window.
Debugging tips
- If samples are wildly unbalanced, check the randomization seed and bucketing (SRM test).
- If significance appears instantly, check for confounded assignment (e.g. geography or browser differences).
- Compare both groups on historical pre-test metrics to sanity-check exchangeability.
- If the treatment group never got the variant (feature flag off), the analysis will show null — verify exposure.
Interview checkpoint
Quick quiz
The key ingredient that lets an A/B test attribute causality is:
A/B tests can be used to measure long-term effects like one-year retention.