Statistics
Intermediate 15 min read Statistics

Hypothesis Testing

Deciding if a difference is really there

Hypothesis testing formalizes 'is this difference real?': null vs alternative hypotheses, test statistics, and p-values that measure evidence against the null — the statistical engine behind A/B tests and analyst verdicts.

null hypothesisalternativep-valueType IType IIt-testsignificance
Prerequisites: Sampling and Inference
Before we start

What you will learn

  • Formulate null and alternative hypotheses
  • Run and interpret common tests (t-test, proportion z-test)
  • Use error rates (Type I/II) to calibrate conclusions
Introduction

What is this?

Hypothesis testing frames claims as opposing pairs: the null H0 (no effect) and the alternative Ha (effect of interest). A test computes how likely the observed data would be under H0; if that probability (p-value) falls below a pre-set α (usually 0.05), you reject H0. The machinery — null, statistic, p-value, Type I/Type II errors — is what makes 'we saw a lift' a defensible claim.

Motivation

Why do you need it?

  • It converts opinion into a falsifiable statement about data.
  • A/B test verdicts, metric regressions and model comparisons all run on it.
  • Understanding error types shows why 'significant' ≠ 'true' and 'not significant' ≠ 'no effect'.
  • Pre-registered tests keep the analysis honest against p-hacking.
Think about it

Mental model

Imagine

A court trial

Think of

Hypothesis testing

The null is 'not guilty'. The prosecution (evidence/data) must be strong enough that 'no effect' is implausible. Failing to prove guilt is not proof of innocence — absence of evidence isn't evidence of absence.

Mechanics

How it works

  • State H0 ('no difference') and Ha ('a difference exists') before collecting data.
  • Pick the test and its statistic (t for means, z for proportions).
  • Under H0, the statistic has a known distribution; compute the observed position.
  • p-value = probability of a statistic as extreme as observed under H0.
  • If p < α, reject H0 (claim evidence for Ha); else fail to reject (inconclusive).

The hypothesis-testing pipeline.

H0 / Ha
Test choice
Compute statistic
p-value
Verdict
Hands-on

Examples

A t-test in scipy

A t-test in scipy
import numpy as np
from scipy import stats
np.random.seed(3)
ctrl = np.random.normal(50, 10, 400)
vari = np.random.normal(53, 10, 400)   # +3 real effect
 
t, p = stats.ttest_ind(vari, ctrl, equal_var=True)
print(f"t={t:.2f} p={p:.5f}")
# p < 0.05 → reject H0, evidence the +3 lift is real (assuming design is sound)

A two-sample t-test compares means; the tiny p-value says random chance producing this gap is unlikely — supporting Ha.

Real world

In production

  • A/B test verdicts: reject H0 to ship the new checkout.
  • Ops metric alerts: 'response time up vs last week' tested before paging.
  • Model comparisons: does version B's MAPE fall beyond noise?
Guidelines

Best practices

  • Fix α and the primary metric BEFORE seeing the sample (pre-registration).
  • Choose tests matching the design: paired data → paired test, independent → two-sample.
  • Report effect size and CIs with the p-value, never alone.
  • Run the 'null is true' simulation (permutation/bootstrap) to sanity-check the test.
Watch out

Common mistakes

  • P-hacking: running many tests until one crosses 0.05.
  • Conflating statistical significance with practical magnitude.
  • Interpreting 'fail to reject' as 'proof of no effect'.
  • Wrong test type (paired vs independent) silently biasing the verdict.
When things break

Debugging tips

  • A test rejects with N=10? Likely assumption violation or tiny data — inspect n and variances.
  • Same numbers, different verdicts across tools? Check equal_var and one/two-tailed settings.
  • p exactly 0.05 feels suspicious — describe the effect with a CI instead of cutoff-gazing.
  • Paired structure present but a two-sample test used — the paired variance is being inflated.
Interviews

Interview checkpoint

Practice

Hands-on lab

Run a Two-Sample Test on Simulated Data
30 min · intermediate
In progress
Objectives
  • Simulate a control and a variant with a known effect
  • Run a t-test and interpret the p-value
  • Show how sample size changes the verdict

Run a Two-Sample Test on Simulated Data

30 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A p-value measures:

2

Failing to reject the null proves the treatment had no effect.

Go deeper

Official documentation

scipy.stats.ttest_ind
docs