EDA
Intermediate 16 min read EDA

Missing Data and Outliers

The two data pathologies worth a decision

Missingness has a mechanism (MCAR/MAR/MNAR) and outliers have a cause (bug or signal) — analyzing both before deciding fixes separates thoughtful pipelines from delete-everything reflexes.

missingnessMCARMARMNARoutlierz-scoreIQR rule
Prerequisites: Univariate Analysis
Before we start

What you will learn

  • Classify missingness mechanisms
  • Detect outliers with z-scores and IQR
  • Choose fixes that respect the cause
Introduction

What is this?

Missing values and outliers are the two pathologies every dataset hides. Missingness falls into mechanisms: MCAR (random, ignorable), MAR (depends on observed variables), MNAR (missing because of the value itself) — each with different fix advice. Outliers are either data bugs (unit error, sensor fail) or real (fraud, whales); classify before removing because the fix differs diametrically.

Motivation

Why do you need it?

  • Deleting MCAR rows is wasteful but safe; deleting MNAR rows biases the sample toward survivors.
  • Imputing with the mean shrinks variance and can hide the signal entirely.
  • Outlier policy is a modeling decision — drop the whales and your revenue model predicts median customers.
  • Interviewers reach for this exact taxonomy to test judgment, not recall.
Think about it

Mental model

Imagine

A diary with missing pages

Think of

Missing data mechanisms

Pages missing at random (MCAR — the wind took some), missing tied to your mood (MAR — you skip entries in bad weeks, a recorded variable), or missing precisely because those were terrible days (MNAR — you burned the pages). The remedy depends on WHY they're gone.

Mechanics

How it works

  • Audit: df.isna().mean().sort_values() for rates; missingness heatmap for patterns.
  • MCAR test: do missing rows differ on observed features? If not, plausibly random.
  • MAR: missingness predicts from other observed columns (or it's close enough to treat that way).
  • MNAR: missing because of the value itself — flag rates and distributions before and after missingness.
  • Outliers: z = |x − mean|/std beyond ~3σ or IQR-based (x < Q1−1.5IQR or > Q3+1.5IQR); then classify bug vs signal.

Decision tree for each pathological column.

Missing?
MCAR/MAR
MNAR
Outlier
Decision
Hands-on

Examples

Detecting and classifying

Detecting and classifying
import pandas as pd, numpy as np
rng = np.random.default_rng(7)
x = np.exp(rng.normal(0, 1, 1000))
x[900] = 5e6                                    # planted outlier
s = pd.Series(x)
q1, q3 = s.quantile(.25), s.quantile(.75)
iqr = q3 - q1
mask = (s < q1 - 1.5*iqr) | (s > q3 + 1.5*iqr)
print(s[mask].tolist()[:5])                      # the massive outlier flagged
print(s.isna().sum())

The IQR rule flags the planted 5e6 cleanly on a skewed column where ±3σ (mean/std) misfires because the tail inflates std. On skewed data the IQR fence is the robust default.

Real world

In production

  • An anomaly team flags transactions beyond k·IQR as candidate fraud, not dropped — they're the signal.
  • A telemetry pipeline marks a sensor's null stack as MNAR and models 'missing' as a state rather than imputing.
  • A churn dataset splits missingness rate by plan — finding MAR prompted including 'has_gap' as a feature.
Guidelines

Best practices

  • Classify every column's missingness mechanism before touching it.
  • On skewed data prefer IQR fences over z-scores.
  • Escalate outlier decisions to stakeholders when rows are few and large.
  • Never drop missing rows wholesale — report how many and why for each column.
Watch out

Common mistakes

  • Imputing MNAR data as if it were random.
  • z-score outlier removal on skewed data (std inflation hides real outliers).
  • Dropping rows with missing target — often the most interesting ones.
  • Removing 'outliers' before the model is built and never documenting the oxbow criteria.
When things break

Debugging tips

  • Compare column distributions with and without missing rows — divergence implies MAR.
  • If z-based outlier removal barely changes anything, the tail was already compressing — use IQR.
  • Recompute missing rates per group; a group-specific rate reveals the mechanism.
Interviews

Interview checkpoint

Practice

Hands-on lab

Classify the Pathologies
30 min · intermediate
In progress
Objectives
  • Audit missing rates and patterns
  • Detect outliers with IQR vs z-score on skewed data
  • Make cause-based fix decisions

Classify the Pathologies

30 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Income column is missing precisely for the highest earners. That's:

2

The IQR-based outlier fence is more robust than ±3σ for skewed columns.

Go deeper

Official documentation

pandas missing-data guide
docs
Missing data mechanisms (Little & Rubin)
docs