Missing Data in EDA
The pattern of absent data is information
Missingness is a pattern, not a bug: examine it with null matrices, per-column budgets and by-group rates before any imputation or deletion decision.
What you will learn
- Map missingness across rows and columns
- Compare missing rates by segment
- Classify missingness pattern for the fix
What is this?
Missing-data EDA studies where, how much, and by whom values are absent: df.isna().sum() gives the per-column budget, a missingness heatmap the row pattern, and grouped rates show whether absence varies by segment. Patterns decide policy — columns missing together get grouped treatment; segment-dependent missingness warns of biased samples.
Why do you need it?
- A column missing in 30% of rows likely means a system, not a typo.
- Missing at higher rates among pro users flags a sampling bias before any test runs.
- Rows whose key fields are blank poison joins silently.
- Seeing missingness as structure turns 'clean it up' into 'handle it correctly'.
Mental model
A survey with blanks in the same questions
Missingness patterns
If every elderly respondent skips the same question, the blank is not random noise — it's a structural gap tied to the segment. Analysts read that pattern the way a survey reviewer reads identical blanks: it names a cause and prescribes a fix.
How it works
- Budget: df.isna().sum() / len(df) per column.
- Rows: df.isna().any(axis=1).sum() and a heatmap of a sample to spot structural blocks.
- By-group: df.groupby('plan').apply(lambda g: g['x'].isna().mean()) shows segment skew.
- Correlated missingness: check if two columns are missing on the same rows.
- Classify: MCAR/MAR/MNAR from the evidence; then choose the handling policy.
Missingness readout.
Examples
Measure, then group
import pandas as pddf = pd.DataFrame({ "plan": ["pro", "basic", "pro", "basic"], "usage": [None, 40, None, 30], "billing": [200, None, 250, 400],})print((df.isna().sum() / len(df)).round(2))print(df.groupby("plan")["usage"].apply(lambda s: s.isna().mean()).round(2))The budget shows usage at 50% missing and billing at 25%; the by-plan group shares tell you WHERE the gaps concentrate.
In production
- An EDA review of an email campaign file finds 'click' missing almost only among unsubscribers — a structural, report-me flag.
- A subscription audit notices billing-amount nulls cluster in legacy plans, proof of a migration gap.
- A feature pipeline flags any column whose null budget breaches 10% as a separate gate.
Best practices
- Compute missingness relative to len, not absolute counts.
- Look for column-pairs missing together — that's a shared cause.
- Segment missingness by key groups; absence is rarely uniform.
- Record the mechanism hypothesis in the EDA findings.
Common mistakes
- Assuming blanks are random because overall rate is low.
- Filling before understanding why — a fill without a mechanism is a guess.
- Ignoring empty-string ('') as a missing-value costume.
- Dropping all null rows and silently changing the population.
Debugging tips
- A column was null for exactly the pre-2025 rows → a migration gap; that's MAR with a timestamp lesson.
- Two columns always null together → a single upstream extract failed twice.
- Missing rate higher by plan → sampling/eligibility bias, handle before any segment comparison.
- Row with all fields missing → an ingestion artifact; inspect source.
Interview checkpoint
Hands-on lab
- Compute per-column null budgets
- Compare null rates across segments
- Write a mechanism guess with policy
Diagnose Missingness by Segment
25 min · intermediate
Quick quiz
A null rate that varies strongly by plan is evidence of:
A 1% overall missing rate guarantees random missingness.