Data Cleaning
Beginner 12 min read Data Cleaning

Data Quality Dimensions

Measure trust before you build on it

Before analysis can be trusted, data must pass six quality gates: accuracy, completeness, consistency, timeliness, validity and uniqueness. Define them, measure them, then clean.

qualityaccuracycompletenessconsistencyvalidityuniquenessfreshness
Prerequisites: Pandas DataFrames
Before we start

What you will learn

  • Name the six quality dimensions
  • Assess a dataset against each one
  • Prioritize cleaning by decision impact
Introduction

What is this?

Data quality measures fitness for use. The classic dimensions: accuracy (right values), completeness (no missing), consistency (same facts agree), timeliness (fresh enough), validity (fits the schema/domain), and uniqueness (no duplicates). An analyst's job starts by scoring each one on real data — then deciding what must be fixed before a decision can rely on it.

Motivation

Why do you need it?

  • Garbage-in resuscitation is the main driver of wrong business decisions.
  • Quality problems compound: one bad join fans out into hundreds of bad rows.
  • Documented quality scores communicate risk to stakeholders who will rely on the analysis.
  • Catching issues at the boundary is 10x cheaper than at the dashboard.
Think about it

Mental model

Imagine

A building inspector's checklist

Think of

Data quality assessment

An inspector doesn't 'clean' the building — they verify foundations, wiring, plumbing, permits, occupancy and duplicates against code, then issue findings. Data quality gates do the same: measure, flag, and only then remediate.

Mechanics

How it works

  • Accuracy: compare a sample against an authoritative source or business rule.
  • Completeness: count nulls per column and empty/zero placeholders.
  • Consistency: cross-check the same fact in two fields or two sources.
  • Timeliness: compare event/load timestamps against the expectation.
  • Validity: check values against allowed domains, formats and ranges.
  • Uniqueness: count exact and key-valued duplicates.
  • Prioritize fixes by the magnitude and direction of decision impact.

Six gates, one assessment pass.

Dataset
Accuracy
Completeness
Consistency
Timeliness
Validity
Uniqueness
Real world

In production

  • A data team publishes a quality scorecard per table so executives see freshness and null rates before relying on KPIs.
  • An ingestion job fails loudly when a core dimension's uniqueness drops below threshold.
  • A vendor evaluation compares competing data providers on completeness and freshness scores.
Guidelines

Best practices

  • Write the expected values down before you look — tests without expectations are decoration.
  • Score each dimension, then repair only what the decision needs.
  • Keep a quality dashboard with time trends so regressions are visible.
  • Document 'known dirty' tables — unsolved issues cost less when they're declared.
Watch out

Common mistakes

  • Cleaning everything 'just in case' and losing traceability.
  • Measuring nulls but not accuracy, then being surprised by wrong values.
  • Fixing data in the notebook without ever restating it upstream.
  • Treating freshness as a binary: 'today's data is clean'.
When things break

Debugging tips

  • When a KPI disagrees with reality, run the six-gate audit — one dimension usually explains it.
  • X-ray two sources of the same fact to expose consistency breaks.
  • Count exact vs fuzzy dupes separately — they fail differently.
  • Re-run the scorecard after a schema change; quality assumptions go stale fast.
Interviews

Interview checkpoint

Practice

Hands-on lab

Score a Sales Export Across Six Gates
30 min · beginner
In progress
Objectives
  • Apply the six dimensions to a messy export
  • Quantify each problem with pandas
  • Publish a findings list, not just fixes

Score a Sales Export Across Six Gates

30 min · beginner

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A date stored as '2026/13/45' fails which dimension?

2

A table with no NULLs is automatically accurate.

Go deeper

Official documentation