Statistics
Advanced 14 min read Statistics

ANOVA and Chi-Square Tests

Comparing many groups without burning the alpha

Beyond two groups: ANOVA compares several means at once; chi-square tests independence in contingency tables. Both need specific assumptions and post-hoc discipline for many groups.

ANOVAchi-squarecontingency tablepost-hoc testscategoricalF-test
Prerequisites: Hypothesis Testing
Before we start

What you will learn

  • Run an ANOVA for 3+ group means
  • Run a chi-square test of independence on categories
  • Apply post-hoc tests and corrections
Introduction

What is this?

When comparisons exceed two groups, separate two-sample tests inflate error. ANOVA partitions variance between-group vs within-group and yields an F-test for 'any difference among means'. Chi-square tests whether rows and columns in a contingency table are independent (segments vs conversion, regions vs plan mix). Both are omnibus tests; follow-ups need post-hoc corrections (Tukey HSD) to identify which groups differ.

Motivation

Why do you need it?

  • Comparing 5 regions with pairwise t-tests is a multiple-testing minefield.
  • Conversion by device/plan/channel questions are contingency-table questions.
  • ANOVA detects 'some difference' cleanly before diving into pairwise comparisons.
  • Chi-square is the workhorse of categorical analytics (funnel stages × variants).
Think about it

Mental model

Imagine

A referee panel judging 5 performances, not just a head-to-head

Think of

ANOVA

You want to know if scores differ at all among the five acts before naming a winner; then a tie-break (post-hoc) says which pairs notably differ. Halt all pairwise comparisons first.

Mechanics

How it works

  • ANOVA: F = between-group variance / within-group variance; large F → some mean differs.
  • Assumptions: roughly normal residuals and similar variances across groups (or Welch's ANOVA).
  • Chi-square: compare observed counts to expected under independence; χ² = Σ(obs−exp)²/exp.
  • Small cell counts break chi-square — use Fisher's exact or simulated p-value.
  • Post-hoc: Tukey HSD controls familywise error across pairwise group means.

Choosing between the omnibus and the pairwise.

3+ groups, continuous
Categories × categories
Significant?
Pairwise details
Hands-on

Examples

ANOVA and chi-square in one block

ANOVA and chi-square in one block
from scipy import stats
import numpy as np
 
# ANOVA: 3 regions, conversion-day metric
rs = [np.random.normal(5, 1, 100) for _ in range(3)]
rs[1] += 0.8   # region 2 genuinely higher
F, p = stats.f_oneway(*rs)
print("ANOVA p:", round(p, 4))
 
# Chi-square: variant × converted
table = np.array([[120, 180],   # control: not-conv, conv
                  [110, 190]])  # variant
chi2, p2, dof, ex = stats.chi2_contingency(table)
print("chi2 p:", round(p2, 4))

f_oneway compares all region means at once; chi2_contingency tests conversion-vs-variant independence in one table.

Real world

In production

  • Comparing click-through across 6 ad creatives with one ANOVA instead of 15 t-tests.
  • Device × conversion tables run through chi-square plus Cramér's V for effect strength.
  • Region × plan mix tables audited for distribution shifts over time.
Guidelines

Best practices

  • Grab the omnibus p-value BEFORE reporting pairwise outcomes.
  • Pair pairwise tests with Tukey/Šidák corrections.
  • Ensure chi-square cells have adequate expected counts (≥5 typically).
  • Report effect sizes (η² for ANOVA, Cramér's V for χ²) beside p-values.
Watch out

Common mistakes

  • Running pairwise t-tests without correction on 6 groups.
  • Chi-square on tiny expected cells producing garbage p.
  • Ignoring unequal variances in ANOVA — use Welch's variant.
  • Significant ANOVA + no post-hoc = 'something differs' but nobody knows what.
When things break

Debugging tips

  • ANOVA significant but no pairwise test is? Run Tukey HSD with the familywise alpha.
  • Chi-square errors on small counts — collapse cells or use Fisher's exact.
  • Region means differ wildly but p is not tiny — sample variance is high; increase n.
  • If one cell of the table dominates χ², that single row is your story; isolate it.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

ANOVA's F statistic compares:

2

Chi-square tests whether two categorical variables are independent.

Go deeper

Official documentation

scipy.stats.f_oneway
docs
scipy.stats.chi2_contingency
docs