Bootstrap and Resampling
Estimate uncertainty by asking the data again and again
Bootstrap resamples your data thousands of times to estimate the uncertainty of any statistic. When the math of standard errors is hard, the computer draws the answer instead.
What you will learn
- Explain what a bootstrap resample is and why it works
- Build bootstrap confidence intervals for means, medians, and ratios
- Use the bootstrap where closed-form formulas break down
What is this?
The bootstrap treats your sample as a stand-in for the population and repeatedly draws new samples from it with replacement. Each resample yields a value of your statistic; the spread of those values approximates the statistic's sampling distribution, giving you a confidence interval without formulas.
Why do you need it?
- It works for any statistic — medians, ratios, quantiles — where standard-error formulas do not exist or are approximations.
- It is a superpower for small, messy samples.
- The same pattern powers permutation tests for comparing groups.
Mental model
Estimating your team's average height by repeatedly re-measuring old measurements
Bootstrap resampling
You are not getting new people; you draw from what you have, and the variation across draws mimics the sampling error.
How it works
- Take the observed sample of size n.
- Draw B resamples of size n, each with replacement (same point allowed multiple times).
- Compute your statistic (mean, median, ratio) on every resample.
- Take the 2.5% and 97.5% percentiles of those B values as the confidence interval.
Many resamples produce a distribution around the statistic.
In production
- Analysts put a confidence band on median order value, which has no tidy formula.
- Product teams bootstrap the LTV uplift from an experiment's payback number.
- Finance bootstraps ratio metrics like LTV:CAC to see how wild the estimate really is.
Best practices
- Use a fixed random seed so your intervals are reproducible.
- Report B (thousands) and the percentile method explicitly.
- Bootstrap can mislead on heavily skewed, tiny samples — pair with the observation that it still has value.
Common mistakes
- Resampling without replacement (which is just the same sample, every time).
- Pooling the samples for a bootstrap instead of resampling each group separately.
- Quoting the mean of the bootstraps as the point estimate instead of the original statistic.
Debugging tips
- If the interval never changes across seeds, check that replacement is actually turned on.
- If the bootstrap standard error is enormous, look at extreme outliers — cap or log them first.
Interview checkpoint
Hands-on lab
- Bootstrap the median order value to get a confidence interval
- Compare the bootstrap CI to a normal-approximation CI
- Observe how the interval narrows as sample size grows
Bootstrap a Confidence Interval
30 min · intermediate
Quick quiz
Bootstrap resampling draws new samples
A bootstrap confidence interval comes from