Feature Engineering
Beginner 14 min read Feature Engineering

Missing Value Imputation

Decide, don't delete

Missingness is information, not just a problem: you choose between dropping, simple imputation, model-based imputation, and explicit missing flags based on the mechanism behind the gaps.

missing valuesimputationMCARMARMNARnanSimpleImputerKNNImputer
Prerequisites: Pandas Indexing and Selection
Before we start

What you will learn

  • Profile missingness as MCAR / MAR / MNAR and choose a strategy per mechanism
  • Implement mean, median, mode, constant, KNN and iterative imputers
  • Decide when to drop rows/columns vs impute, and when a missing-flag column helps
Introduction

What is this?

Missing value imputation replaces absent values so downstream models see a complete matrix. The choice of method depends on the mechanism: MCAR (missing completely at random), MAR (missing at random given observed predictors), or MNAR (missing related to the value itself). Dropping is sometimes right: when missingness is tiny (<5%) and MCAR, or when a column is mostly empty.

Motivation

Why do you need it?

  • Most model classes and many libraries reject NaN directly; a complete matrix is a hard prerequisite.
  • Imputation keeps the row's other (informative) columns — dropping whole rows wastes signal.
  • A missing-flag encodes the 'something unusual happened here' signal that imputation alone erases.
  • The mechanism decides the method: MCAR is safe to drop, MNAR demands domain reasoning.
Think about it

Mental model

Imagine

Filling a pothole you haven't inspected

Think of

Blind mean-imputation

If the pothole hides a broken pipe, filling it with gravel is worse than flagging it. Missingness mechanics tell you whether gravel, a patch, or a yellow flag is the right move.

Mechanics

How it works

  • Profile: df.isna().sum() per column, plus by-group missingness to spot MAR patterns.
  • Mechanism guess: if missingness correlates with observed features → MAR; if it flags extreme values → MNAR.
  • Pick method: drop (tiny MCAR), constant/mean/median (low missingness), KNN/iterative (arbitrary patterns), model-based (rich observed predictors).
  • For trees, NaN can be handled natively (sklearn HistGradientBoosting); for linear models and neural nets, impute.
  • Evaluate imputation at model level: score with and without, on the validation fold — not on imputation-error alone.

Choosing an imputation strategy.

Profile
Mechanism
Strategy
Flag
Validate
Hands-on

Examples

SimpleImputer with a missing-flag

SimpleImputer with a missing-flag
import pandas as pd
import numpy as np
from sklearn.impute import SimpleImputer, KNNImputer
 
df = pd.DataFrame({
    "age": [34, 28, np.nan, 45, np.nan, 52],
    "income": [70_000, 88_000, 120_000, np.nan, 90_000, 84_000],
})
 
imputer_mean = SimpleImputer(strategy="median")
arr = imputer_mean.fit_transform(df)
df_filled = pd.DataFrame(arr, columns=df.columns)
 
df_filled["age_missing"] = df["age"].isna().astype(int)
df_filled["income_missing"] = df["income"].isna().astype(int)
 
print(df_filled)
print("imputer medians:", imputer_mean.statistics_)
 
knn = KNNImputer(n_neighbors=2)
print("
KNN result:")
print(knn.fit_transform(df))

The median imputer fills NaN with learned medians, and separate is_missing flags preserve the fact that a value was absent — a signal some models should learn from.

Real world

In production

  • Credit scoring: income missingness strongly predicts default risk (MNAR-ish) — keep the flag, impute conservatively.
  • IOT sensor floods: impute last-observed-value; missingness counts feed a separate 'signal quality' model.
  • Healthcare: MAR missingness driven by lab-ordering patterns; use MICE/iterative with domain priors.
Guidelines

Best practices

  • Impute with the mechanism in mind, not habit; impute median for skewed columns, mean for symmetric ones; fit the imputer on train folds only; add flags when missingness exceeds ~5% or the mechanism is MNAR; re-run imputation checks after every ETL change.
Watch out

Common mistakes

  • Using the column mean on heavily skewed data; imputing with the global mean per-fold (should be fold-local); deleting rows when other columns are intact; assuming all NaN are equal (None vs empty-string vs 0 are different bugs); imputing test data with the test column's own statistics.
When things break

Debugging tips

  • Compare missingness percentage train vs prod — drift here is a pipeline bug, not a modeling tweak.
  • Plot a heatmap of column-pair missingness to spot shared-cause gaps.
  • After imputation confirm no NaN survived: df_filled.isna().sum().sum() == 0.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which missingness mechanism implies the gap is related to the unobserved value itself?

2

Your income column is 40% missing. Best first move?

Go deeper

Official documentation

sklearn.impute
docs
Missing Data: A Conceptual Review - PubMed
video
pandas: working with missing data
docs