Handling Missing Data
Absence is a datum, not a blank
The absence of a value carries meaning. Detect it, model why it's missing, and handle it deliberately — drop, fill, flag or encode — so absence never silently skews results.
What you will learn
- Quantify missingness by column and pattern
- Classify the missingness mechanism
- Apply drop/fill/flag decisions with reason
What is this?
Handling missing data begins with measuring and ends with a defensible policy. Counts by column, missingness patterns (rows missing together), and mechanism guesses (missing at random vs structural) drive the choice: drop rows when the critical column must exist, fill for residuals, add an is_missing flag when absence is meaningful, or leave deliberate NA for domain-aware consumers.
Why do you need it?
- Every aggregate that silently skips NaN risks a quietly wrong number.
- Missingness is often the signal: 'no payment record' is a churn signal.
- Naive fills (zeros, means) bias downstream models and reports.
- A documented missing-data policy is what separates senior from junior analysis.
Mental model
A meal ticket with dishes crossed out
Missing data handling
The kitchen must know whether 'absent' means sold-out (fill with substitute), it was never offered (leave off), or there was a data entry error (flag and retry). Handling missing data is choosing, per reason, what the ticket should say.
How it works
- Count: df.isna().sum(); pattern: sort by null columns or df[df.isna().any(axis=1)].
- Mechanism: MCAR (random), MAR (related to a measured var), MNAR (depends on the missing value itself).
- dropna(subset=[...]) removes rows missing decision-critical fields.
- fillna with domain defaults, medians per group, or ffill/bfill for ordered data.
- Add is_missing columns so models and reports can consume absence as information.
- Choose and document — the policy ships with the pipeline.
The missing-data policy.
Examples
A policy in code
import pandas as pddf = pd.DataFrame({"sales": [100, None, 80, None], "qty": [1, 2, None, 4]})print(df.isna().sum())df["sales_missing"] = df["sales"].isna().astype(int)clean = df.dropna(subset=["sales"]).fillna({"qty": 0})print(clean)Counts first; then the decision-critical sales column drops its missing rows, qty fills with a justified zero, and the flag preserves the absence signal.
In production
- A retention model keeps is_missing_activity as an explicit churn feature.
- A billing audit drops rows missing invoice_id but never imputes money.
- A demand pipeline forward-fills gaps in ordered time series before resampling.
Best practices
- Never let pandas' default skipna make your missing-data decision silently.
- Fill only where the domain justifies the value; otherwise flag and carry NA.
- For time series, ffill/bfill respects ordering; median fills are for cross-sectional columns.
- Re-report null counts after cleaning so the shrink is visible.
Common mistakes
- Filling everything with zeros and corrupting averages.
- Dropping missing rows and forgetting the population changed.
- Imputing money columns where absence means 'not applicable'.
- Treating '' (empty string) as not missing — it's a missing value wearing text.
Debugging tips
- An odd mean → count nulls; zeros may be axis of the problem.
- fillna(0) hiding an upstream field that never arrived — check timeliness, not nulls.
- ddffill in a sorted frame is safe; unsorted it fabricates history.
- If a join 'loses rows', it's a null/empty key on one side.
Interview checkpoint
Hands-on lab
- Measure nulls by column and pattern
- Classify columns as drop/fill/flag
- Implement and document the policy
Write a Defensible Fill Policy
30 min · beginner
Quick quiz
The most honest summary companion to 'filled missing sales with 0' is:
Forward-fill for ordered data is safe on unsorted frames.