Building a Cleaning Pipeline
From one-off script to trusted product
Cleaning is a repeatable stage, not a notebook adventure. Compose small pure functions, run quality gates, keep a log, and make the whole thing idempotent and testable.
What you will learn
- Structure cleaning as composable stages
- Add quality gates between stages
- Make the pipeline idempotent and observable
What is this?
A cleaning pipeline is a sequence of small, pure, testable stages — normalize, coerce, dedupe, validate — each asserting a boundary contract before passing the frame on. A log records what changed at every stage, and the design guarantees the same inputs produce the same outputs no matter how many times it runs (idempotence).
Why do you need it?
- One-off cleaning untraceable in a notebook dies the day the data changes.
- Composed stages are testable, reviewable, and reusable across reports.
- Quality gates catch regressions the instant a source renames a column.
- Idempotent pipelines can be rerun safely on schedule and on demand.
Mental model
An industrial prep line with inspection stations
A cleaning pipeline
Each conveyor stage does one job — wash, sort, weigh, pack — and a scale at each station refuses underweight fruit (the gate). The line is documented, repeatable, and a forklift rerun doesn't double-kill any fruit because nothing is stateful. Pipelines should run exactly that way.
How it works
- Each stage = one pure function (frame in, frame out) with a named transform.
- Gates between stages assert shape, dtypes, null budgets and key uniqueness.
- A run log (per-stage row counts, dropped counts, coerced counts) ships with output.
- Idempotence: raw inputs never mutated; outputs written to stamped paths.
- Testability: hand-built fixtures validate each stage before it touches production data.
- Fail loudly at gates; degrade honestly (flag, not guess) elsewhere.
Stages, gates, log.
Examples
One stage, gated and pure
import pandas as pd def coerce_money(frame: pd.DataFrame, col: str) -> pd.DataFrame: out = frame.copy() cleaned = out[col].str.replace(r"[$,]", "", regex=True) out[col] = pd.to_numeric(cleaned, errors="coerce") out[col + "_missing"] = out[col].isna().astype(int) out[col] = out[col].fillna(0) return out df = pd.DataFrame({"amt": ["$10", "20", "n/a"]})df = coerce_money(df, "amt")print(df)A pure stage returns a new frame, flags coercion failures, and lets the gate upstream see exactly what changed.
In production
- A nightly revenue pipeline runs ingest → clean → validate → write with a unit-tested stage each.
- An orchestrator reruns a failed batch idempotently after a source fix.
- A compliance team reads the run log to certify which transformations ran on which day.
Best practices
- One transformation per function; compose, don't chain 30 ops in a cell.
- Assert at stage boundaries, not between every line.
- Log counts and dropped rows; numbers change and someone must see it.
- Fixtures with known answers make each stage's tests obvious.
Common mistakes
- Mutating the source frame inside a stage, breaking reruns.
- Gates so strict they trip on healthy variance, or so lax they never catch schema drift.
- Cleaning steps buried in plotting notebooks where no test can reach them.
- Skipping the log — 'why is the number different?' becomes archaeology.
Debugging tips
- Replay one failed stage on its fixture first — stage isolation shrinks the bug hunt.
- A gate tripping on rename means the contract is doing its job; update the contract.
- Comparing 'raw vs clean' totals exposes which stage altered the population.
- When reruns drift, hunt unsampled ops or row-order dependence — idempotence kills both.
Interview checkpoint
Quick quiz
Idempotence means a pipeline:
A gate checking null budget belongs between pipeline stages.