Statistics
Intermediate 13 min read Statistics

Confidence Intervals

Estimate ± uncertainty, with coverage guarantees

A confidence interval is a range built from a sample that captures the population parameter with a stated probability over repeated sampling — the honest replacement for a naked point estimate.

confidence intervalmargin of error95%bootstrapstandard error
Prerequisites: Sampling Distributions
Before we start

What you will learn

  • Build a mean CI from SE and t/z critical values
  • Explain the repeated-sampling definition of coverage
  • Compare parametric and bootstrap intervals
Introduction

What is this?

A confidence interval (CI) for a parameter is an interval computed from a sample with a coverage guarantee: if you repeated the experiment many times, 95% of the intervals would contain the true value. It translates standard error into a usable range: estimate ± critical·SE. It's not the probability the true value is inside THIS interval — it's a statement about the procedure.

Motivation

Why do you need it?

  • Point estimates hide all the uncertainty that decisions need.
  • A wide CI exposes tiny samples or noisy metrics before you bet on them.
  • A/B differences are reported with CIs on the lift, showing both size and precision.
  • Bootstrap CIs rescue you when normality is dubious.
Think about it

Mental model

Imagine

A survey saying '49% ± 3%'

Think of

Confidence interval

The ±3% (margin of error) is built from the standard error. '±3%' only means the sampling procedure is calibrated, not that the true number is 'somewhere in there with 95% probability given this one poll'.

Mechanics

How it works

  • Estimate the parameter (e.g., sample mean ȳ).
  • Compute SE = σ/√n (estimate σ from s).
  • Multiply by the critical value: 1.96 (z at 95%) or a t quantile for small n.
  • Interval = ȳ ± z·SE. In 95% of repeated experiments the true μ lies inside.
  • Bootstrap CIs instead use the empirical percentiles of resampled statistics.

Coverage in action.

Population μ
Repeat sampling
Each yields a CI
Coverage
Report one
Hands-on

Examples

Parametric and bootstrap 95% CIs

Parametric and bootstrap 95% CIs
import numpy as np
rng = np.random.default_rng(5)
data = rng.normal(50, 8, 300)
 
se = data.std(ddof=1) / np.sqrt(len(data))      # standard error
ci = (data.mean() - 1.96*se, data.mean() + 1.96*se)
print(tuple(round(v,2) for v in ci))            # z-based CI
 
boot = np.array([data[rng.integers(0,300,300)].mean() for _ in range(2000)])
print(np.percentile(boot, [2.5, 97.5]))          # bootstrap CI

The z-interval assumes normality of the mean (fine here); the bootstrap percentiles make the same 95% claim without that assumption. For a normal mean they nearly coincide.

Real world

In production

  • A pricing analytics dashboard renders every KPI as 'current ± 95% CI', letting PMs spot real drift.
  • An experimentation platform shows CI on revenue-per-user lift, grounding rollouts in precision.
  • A feature team bootstraps CI on AUC differences so 'we're better' is a defensible claim.
Guidelines

Best practices

  • Pair every estimate with its CI; a bar chart without error bars hides powerlessness.
  • Choose CI width by decision risk — 99% for damage-prone releases, 90% for scouting metrics.
  • Use bootstrap for skewed or exotic statistics.
Watch out

Common mistakes

  • Saying 'there's a 95% probability that μ is in this interval' — the parameter is fixed; the interval is random.
  • Using z=1.96 for tiny samples where the t-critical is larger.
  • Comparing CIs by eyeballing overlap instead of computing differences and their CI.
When things break

Debugging tips

  • If the CI is shockingly wide, check n and the variance estimate: SE = s/√n explains both.
  • For asymmetric distributions, a symmetric ±CI misleads; bootstrap gives the skew.
  • Recompute with the t-distribution at small n — z-based is too optimistic.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Doubling sample size narrows a z-based mean CI by a factor of:

2

The 95% CI contains the parameter with 95% probability for a single observed interval.

Go deeper

Official documentation

scipy.stats.t confidence interval example
docs