Statistics
Intermediate 14 min read Statistics

Sampling and Inference

From a sample to confident claims

Inference generalizes from a sample to the population. Random, representative sampling and a known standard error turn a sample estimate into a statement with measurable uncertainty.

samplinginferencestandard errorrandom samplepopulationrepresentative
Prerequisites: Distributions
Before we start

What you will learn

  • Design random, representative samples
  • Compute and interpret the standard error
  • Generalize only as far as the sampling supports
Introduction

What is this?

Inference uses a sample to estimate a population parameter with quantified uncertainty. A random sample lets chance do the work: each unit has a known chance of inclusion, so the sample mean concentrates near the population mean as n grows. The standard error (SD/√n) measures that concentration; representative and non-absorbing designs keep the estimate honest.

Motivation

Why do you need it?

  • You rarely have the population — you sample events, customers, sessions.
  • Sampling bias (only active users, only logged-in) silently invalidates conclusions.
  • The standard error is precisely what confidence intervals and tests are built on.
  • Understandable inference keeps analysts from declaring 100% certainty from an n=40 slice.
Think about it

Mental model

Imagine

Tasting one spoonful of soup

Think of

Sampling

A well-stirred pot (population) lets one spoonful (random sample) represent flavor. Taste the top cream only (bias) and you misjudge the whole pot — sample design is about how you stir.

Mechanics

How it works

  • Define the population and the sampling frame (the actual list you sample from).
  • Draw randomly with known inclusion, ideally stratified/simple random sampling.
  • Estimate the parameter (mean, proportion) from the sample.
  • Standard error SE = σ/√n (population) or s/√n (sample SD), shrinking as n grows.
  • Use SE to build confidence intervals and compare claims with uncertainty stated.

Sampling to inference.

Population
Frame
Random sample
Estimate
Inference
Hands-on

Examples

Standard error in action

Standard error in action
import numpy as np
np.random.seed(2)
pop = np.random.normal(100, 15, 1_000_000)   # true population
for n in (30, 300, 3000):
    sample = np.random.choice(pop, n)
    se = sample.std(ddof=1) / np.sqrt(n)
    print(f"n={n:5d} mean={sample.mean():6.2f} se={se:.3f}")

Bigger samples tighten the estimate: SE shrinks like 1/√n, so quadrupling n halves the uncertainty.

Real world

In production

  • Survey estimates of CSAT with margin of error reported, not just the score.
  • Sampling 10k sessions to measure page performance with a stated CI.
  • Audit sampling over claims to certify accuracy without every row.
Guidelines

Best practices

  • Randomly sample from the intended population, never the convenient subset.
  • Report n, SE/margin of error and the inclusion mechanism with every estimate.
  • Stratify on important groups (regions, plans) when group sizes vary.
  • When available, use the whole population — sampling is for when you can't.
Watch out

Common mistakes

  • Sampling from a self-selected or convenient slice and generalizing anyway.
  • Ignoring the finite overlap between frame and population.
  • Reporting a sample mean with zero uncertainty.
  • Using sample SD where population math was implied and vice versa.
When things break

Debugging tips

  • Strange point estimate? Check the frame: did 'population' silently mean 'active'?
  • SE too small to believe? Confirm n is actual observations, not duplicate rows.
  • Results don't reproduce — sample too small or too biased; check inclusion rules.
  • Compare your sample's means on known columns to the population to catch bias.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The standard error shrinks with sample size in proportion to:

2

A self-selected poll sample is a valid random sample of the population.

Go deeper

Official documentation

NIST — sample and population
docs