Missing Data
Decide, don't default, when values are absent
NaN, None, and sentinel values live in every real dataset. The discipline is to detect missingness, measure it, and decide deliberately — drop, fill, flag, or encode — rather than let pandas guess.
What you will learn
- Detect and count missing values precisely
- Choose drop vs fill vs flag per context
- Keep missingness patterns visible in analysis
What is this?
pandas represents absence with NaN (float 'not a number') and honours None equivalently in detection. isna()/isnull() build masks; dropna() removes them; fillna() imputes; and missing can also be flagged as its own category. Which choice is right depends on why the data is missing — that's the analytics decision.
Why do you need it?
- Silent pandas operations (sum, mean) skip NaN without telling you.
- Missingness is often informative: 'no transaction' is different from '0'.
- Naive fills (zeros/means) bias every downstream aggregate.
- A defensible missing-data story is a core sign of a senior analyst.
Mental model
A waiter marking items as '86'd' (out) on a ticket
Missing data handling
The kitchen needs to know an item is unavailable — and why. Just serving '0' instead of noting the absence hides the truth. dropna (leave it off the ticket), fillna(0) (say it cost nothing), flag (mark '86'd') — each tells the kitchen something different.
How it works
- df.isna().sum() counts nulls per column; df.isna().any(axis=1) flags rows.
- df.null() checks are the same as isna(); == NaN fails (NaN != NaN).
- df.dropna(subset=['col']) removes rows missing a required column.
- df.fillna({'sales': 0, 'plan': 'unknown'}) fills different defaults per column.
- Forward/backward fill (method='ffill'/'bfill') propagates the last seen value for ordered data.
- Missingness as category: skipna=False in describe, or adding is_missing columns, keeps the signal visible.
The missing-data decision tree.
Examples
Measure, then choose
import pandas as pddf = pd.DataFrame({"sales": [100, None, 80, None], "qty": [1, 2, None, 4]})print(df.isna().sum())print(df.dropna(subset=["sales"])) # keep the decision-column rowsdf["sales_missing"] = df["sales"].isna().astype(int)print(df.fillna({"sales": 0})) # explicit zero only for salesCount the gaps, drop only on the critical column, keep a missingness flag, and fill with an explicit and defensible default.
In production
- A retention report treats 'no login event' as a missing flag, not a zero, preserving churn signal.
- An ML pipeline imputes medians per group (plan) and adds an is_missing feature.
- A billing audit drops rows missing invoice_id but never imputes money columns.
Best practices
- Count missingness before any aggregate — the summary already ignoring nulls is the trap.
- Fill with domain meaning (0 for counts, 'unknown' for categories, median for scale features).
- Prefer a missing flag over silent drop when the absence is meaningful.
- For ordered time data, ffill/bfill is often the right temporal imputation.
Common mistakes
- Using == None or == NaN in filters — matches nothing.
- Filling everything with 0 and inventing false low averages.
- dropping rows and forgetting that the population shrunk.
- aggregating with a mix of default skipna behavior and surprised at skimpy counts.
Debugging tips
- A mean that looks too low → count nulls first; zeros may be phantom.
- A join result that 'lost rows' → NaN keys on one side, likely from empty strings.
- fillna with method='ffill' on unsorted data propagates across wrong regions — sort first.
- describe() shows 'count < length', your instant missingness meter.
Interview checkpoint
Quick quiz
Which correctly counts nulls per column?
The pandas .sum() on a column skips NaN by default.