Statistics
Intermediate 15 min read Statistics

Sampling and Sampling Distributions

How random noise in your data becomes error in your estimates

Sampling turns a population into a tractable dataset, but the sample is random: the sampling distribution of a statistic quantifies how much an estimate bounces around the true value.

samplingstandard errorsample meanrandom samplebootstrapsampling distribution
Prerequisites: Normal Distribution
Before we start

What you will learn

  • Explain why sample statistics vary across samples
  • Define the standard error and its formula
  • Use bootstrap resampling to estimate uncertainty
Introduction

What is this?

A sample is a subset of the population you actually measure. A statistic (mean, median, model coefficient) computed from a sample is a random variable — draw a different sample and the number changes. The sampling distribution of a statistic is that distribution over all possible samples, and its standard deviation (the standard error) is the margin of error attached to any estimate.

Motivation

Why do you need it?

  • Every experiment is a sample: A/B metrics bounce even with no real effect.
  • The standard error tells you how much uncertainty to attach to a reported mean or AUC.
  • Bootstrap resampling estimates uncertainty without normality assumptions.
  • Model comparison without sampling error awareness is a classic false-positive machine.
Think about it

Mental model

Imagine

Scooping soup from a pot and tasting one spoonful

Think of

Estimating a population from one sample

Each spoonful differs slightly (sampling variability); averaging many spoonfuls is stable (central limit). How much one spoonful wanders is the standard error.

Mechanics

How it works

  • A random sample picks observations so every member has equal inclusion probability.
  • The sample mean ȳ estimates the population mean μ; its sampling distribution has mean μ and standard error σ/√n.
  • Standard error shrinks only with √n — quadruple the sample to halve the error.
  • The CLT makes ȳ approximately normal for large n even when the population is skewed.
  • Bootstrap: resample your data with replacement many times, recompute the statistic each time, and look at the spread of the recomputed values.

From population to estimated ± error.

Population
Random sample
Estimate ȳ
SE = σ/√n
Report
Hands-on

Examples

Bootstrap standard error in Python

Bootstrap standard error in Python
import numpy as np
rng = np.random.default_rng(0)
data = rng.exponential(scale=5, size=500)     # skewed population
 
boot_means = [data[rng.integers(0, 500, 500)].mean() for _ in range(2000)]
se = np.std(boot_means)                        # bootstrap SE
print(round(data.mean(), 2), round(se, 3))
print(np.percentile(boot_means, [2.5, 97.5]))  # bootstrap CI

We resample the observed data with replacement 2000 times; the spread of the 2000 means is the bootstrap standard error, and its percentiles give a distribution-free confidence interval — no normality required.

Real world

In production

  • An A/B team reports conversion differences with bootstrap CIs for non-normal revenue metrics.
  • A pricing team sizes a study: SE = σ/√n tells them how many users yield a detectable change.
  • An ML platform worker bootstraps AUC differences to decide whether model A really beats model B.
Guidelines

Best practices

  • Report estimates with their standard errors, not naked point values.
  • Size studies using SE = σ/√n before collecting data.
  • Use bootstrap when the statistic is exotic (median, quantile, metric difference) or data is skewed.
Watch out

Common mistakes

  • Treating the sample as the population — quoting the sample mean with zero error.
  • Ignoring that SE shrinks as √n, so 'just add more data' has diminishing returns.
  • Bootstrap on tiny samples (n < 30) or without replacement (that's the original sample).
When things break

Debugging tips

  • If bootstrap CI doesn't contain the observed estimate, you resampled wrong (no replacement).
  • SE too big to be useful? The remedy is n, not a different formula.
  • Compare bootstrap CI vs z-based CI; big disagreement flags skew or small n.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

To halve the standard error, you must multiply the sample size by:

2

Bootstrap estimates sampling variability by resampling with replacement.

Go deeper

Official documentation

Resampling Methods — scikit-learn
docs
Sampling distribution — Wikipedia
docs