Statistics
Intermediate 18 min read Statistics

Hypothesis Testing

Deciding with uncertainty

Hypothesis testing formalizes 'is this difference real?': null hypothesis, test statistic, p-value, Type I/II errors, and choosing the right test for the data.

null hypothesisp-valuet-testchi-squaretype 1 errortype 2 errorpower
Prerequisites: Sampling Distributions
Before we start

What you will learn

  • Set up null and alternative hypotheses
  • Run a t-test and interpret its p-value
  • Distinguish Type I and Type II errors and their costs
Introduction

What is this?

Hypothesis testing begins with a null hypothesis H₀ (no effect) and an alternative H₁ (effect). You compute a test statistic, see how extreme it is under H₀, and convert that extremeness into a p-value. If the p-value is below the significance level α (usually .05), you reject H₀. The machinery — t-test, chi-square, ANOVA — is the same decision procedure in different data shapes.

Motivation

Why do you need it?

  • Product decisions need a principled answer to 'is this metric change real or noise?'.
  • Model comparison uses the same logic when you ask if AUC improvements are chance.
  • Choosing the wrong test invalidates the whole decision.
  • Understanding Type I/II errors tells you how many false alarms and missed effects you tolerate.
Think about it

Mental model

Imagine

A court trial

Think of

Hypothesis testing

H₀ is 'innocent until proven guilty'. Evidence (the test statistic) must reach a high bar (threshold α) to convict. Decline to convict ≠ proof of innocence; failure to reject H₀ isn't proof of no effect.

Mechanics

How it works

  • State H₀ (no effect) and H₁ (effect).
  • Pick a test whose statistic has a known null distribution (t, χ², F).
  • Compute the statistic and its sampling distribution location under H₀.
  • p-value = probability of a statistic this extreme (or more) if H₀ is true.
  • Reject H₀ when p < α; otherwise fail to reject. α is the Type I error you accept.

A two-sample t-test decision.

Two groups
t = Δ/se(Δ)
Null distribution
p-value
Decision
Hands-on

Examples

A two-sample t-test in scipy

A two-sample t-test in scipy
import numpy as np
from scipy import stats
rng = np.random.default_rng(4)
ctrl = rng.normal(1.0, 0.4, 500)
trt  = rng.normal(1.12, 0.4, 500)      # +12% shift
t_stat, p = stats.ttest_ind(trt, ctrl)
print(round(t_stat, 3), p)             # large |t|, p ~= 0.000004
# p < 0.05 → reject H0: difference is statistically significant

The t-statistic standardizes the mean difference; its p-value is tiny because a difference this big is implausible under 'no effect'. Note the difference is also economically small — statistical significance ≠ practical importance.

Real world

In production

  • A/B tests on checkout buttons use t/z-tests on conversion differences with α set per product risk.
  • A data-science team runs chi-square tests to see if churn associates with plan tiers.
  • An anomaly system runs a quick Welch's t-test on current-vs-baseline latency before paging.
Guidelines

Best practices

  • Pre-register the hypothesis and α before looking at data.
  • Choose the test to match the data: t for means, chi-square for counts, ANOVA for 3+ groups.
  • Report effect size and confidence interval, not just a p-value.
  • Set α from the cost of a false alarm; relax α when Type II error is costlier.
Watch out

Common mistakes

  • p-hacking: running 20 tests and cherry-picking the one below .05.
  • Drawing the 'fail to reject' = 'no effect' conclusion — absence of evidence.
  • Using a t-test on heavily skewed data or count data without transformation.
  • Queueing on sample size as 'small' while ignoring power entirely.
When things break

Debugging tips

  • If p ≈ 1 in a t-test, you may have the groups reversed or signs flipped.
  • Check equal-variance assumptions; use ttest_ind(equal_var=False) for unequal variance.
  • Recompute p from scratch with a permutation test when distribution assumptions feel shaky.
Interviews

Interview checkpoint

Practice

Hands-on lab

Run, Interpret and Power an A/B Test
30 min · intermediate
In progress
Objectives
  • Run a two-sample t-test and read its p-value
  • Demonstrate Type I error rate empirically
  • Show how sample size and effect size change power

Run, Interpret and Power an A/B Test

30 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A p-value of 0.03 at α = 0.05 means you:

2

Failing to reject the null hypothesis proves the null is true.

Go deeper

Official documentation

scipy.stats.ttest_ind
docs
Hypothesis testing — Wikipedia
docs