Pandas
Intermediate 16 min read Pandas

Data Cleaning

The unglamorous majority of the job

Tidy data, dtype fixing, missing values, duplicates and inconsistent strings: cleaning is 60% of real analysis, and pandas gives systematic tools for each.

cleaningmissingduplicatesdtypetidy dataisinstr accessor
Prerequisites: Indexing and Selection
Before we start

What you will learn

  • Detect and repair dtype issues
  • Handle missing values deliberately
  • Dedupe and standardize strings
Introduction

What is this?

Data cleaning means making a dataset faithful to its semantics: correct dtypes, no spurious strings where numbers belong, missing values represented and handled deliberately (not dropped blindly), duplicates removed, and consistent categories. pandas delivers a method per symptom — .isna, .fillna/.dropna, .drop_duplicates, .astype, .str accessor, .isin.

Motivation

Why do you need it?

  • Bad data quietly becomes bad features, bad predictions and bad revenue reports.
  • ML models choke on mismatched types or NaN hybrids.
  • A StandardScaler on object columns silently fails or coerces.
  • Cleaning is where the standard deviation that makes models trustworthy lives.
Think about it

Mental model

Imagine

Prepping ingredients before cooking

Think of

Cleaning before analysis

You don't peel, seed or sort your vegetables mid-boil; the prep table is where consistency is ensured. Data cleaning is that prep — messy inputs become uniform, cookable ingredients.

Mechanics

How it works

  • df.dtypes + df.isna().sum() audit types and holes.
  • Coerce with pd.to_numeric(..., errors='coerce') and .astype after inspecting unique().
  • Missing handling: fillna(value/strategy), dropna(how/threshold/subset), or leave NA for imputers.
  • .drop_duplicates(subset=keys) dedupes records by signature columns.
  • String hygiene via the .str accessor: strip(), lower(), replace(), and .replace / .map for mapping labels.

A cleaning sequence.

Audit
Fix types
Handle NA
Dedupe
Standardize
Hands-on

Examples

A typical clean in six lines

A typical clean in six lines
import pandas as pd
df = pd.DataFrame({
  "user": ["a", " a ", np.nan, "b"],
  "rev":  ["10", "20", "30", "not-a-number"],
})
df["user"] = df["user"].str.strip().replace("", None)
df["rev"] = pd.to_numeric(df["rev"], errors="coerce")
df = df.dropna(subset=["rev"])
df = df.drop_duplicates(subset=["user"])
print(df)

strip() killed the whitespace, to_numeric coerced 'not-a-number' to NaN and dropna removed it, and ' a ' deduped to one row. Each step was deliberate and visible.

Real world

In production

  • A nightly sales load runs a cleaning module that types, fills and dedupes before the warehouse model.
  • A churn model checks isna().sum() per column monthly, alerting when null rate drifts above SLA.
  • An NLP pipeline lowercases and strips user-generated text before encoding.
Guidelines

Best practices

  • Audit before and after cleaning: dtypes, nulls, counts.
  • Never drop NAs 'to be safe' — know which column's missingness is informative.
  • Keep a RAW copy during exploration.
  • Make cleaning a function you can rerun on fresh data, not notebook cells.
Watch out

Common mistakes

  • Running .dropna() and erasing whole rows that had one irrelevant null.
  • Silently coercing strings to numbers and losing 'N/A vs 0 vs '' meanings.
  • Forgetting to strip whitespace so 'a' and ' a ' became separate categories.
  • Dropping duplicates on all columns instead of the business key.
When things break

Debugging tips

  • df['col'].value_counts(dropna=False) exposes the 'N/A' text traps.
  • isna().sum().sort_values() ranks the holes.
  • If a numeric column comes back object, df['col'].apply(type).value_counts() shows the renegade types.
Interviews

Interview checkpoint

Practice

Hands-on lab

Clean a Real-Smelling Messy CSV
30 min · intermediate
In progress
Objectives
  • Detect dtype damage and repair it
  • Handle missing values deliberately
  • Dedupe and standardize text

Clean a Real-Smelling Messy CSV

30 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

pd.to_numeric(...errors='coerce') turns unusable strings into:

2

Duplicated rows are always identical across every column.

Go deeper

Official documentation

Working with missing data
docs
pandas string methods
docs