Data Engineering Foundations
Intermediate 14 min read Data Engineering Foundations

Data Quality Foundations

Trust you can measure, gates you can ship

Data quality is measured: completeness, uniqueness, timeliness, validity, consistency and accuracy. Engineering data quality means asserting those dimensions with automated checks that gate pipelines.

data qualitydimensionsassertionsexpectationsSLAgates
Prerequisites: Data Pipeline Fundamentals Data Governance Basics
Before we start

What you will learn

  • List the six data-quality dimensions with a concrete example each
  • Write assertions (row counts, freshness, uniqueness) in a pipeline
  • Design gates so bad data stops before consumers see it
Introduction

What is this?

Data quality is a set of measurable properties of a dataset: completeness (all expected records), uniqueness (no duplicates), timeliness (freshness vs SLA), validity (values conform to rules), consistency (same thing doesn't mean different things in different tables) and accuracy (values reflect reality). Data engineering encodes these as automated assertions that run with every pipeline and gate downstream loads.

Motivation

Why do you need it?

  • Bad data silently misleads dashboards, models and legal decisions — and it's far more expensive to fix after consumption.
  • Automated gates turn 'we hope it's fine' into 'the pipeline refuses to publish wrong data'.
  • Measurable dimensions give teams a shared scorecard instead of vibes.
  • Quality work compounds: gating early rows today prevents modeling bugs later.
Think about it

Mental model

Imagine

An assembly-line inspection checkpoint

Think of

A data-quality gate

Cars are inspected at each station — if the wheel alignment fails, the car doesn't roll on. Data should be gated the same way: don't publish a table that failed its checks.

Mechanics

How it works

  • Pick dimensions that matter per dataset (not every dimension for every table).
  • Write assertions: row count within [min,max], primary key unique, freshness lag < SLA, values in valid set, schema matches contract.
  • Run assertions as part of the pipeline (dbt tests, Great Expectations, Soda) so failures abort the run.
  • Tier the response: hard gate (block publish), soft gate (alert) or quarantine (park bad rows).
  • Track quality history and SLAs in dashboards; escalate repeat offenders to owners.

Quality dimensions mapped to pipeline gates.

Raw load
Transform
Publish model
Consumers
Real world

In production

  • A nightly load fails the gate when TOTAL=0 or when 2% of expected stores are missing (completeness).
  • A 'customer' table refuses to publish duplicates on the natural key (uniqueness).
  • A streaming KPI dashboard pages when log_minutes_lag exceeds the SLA (timeliness).
  • A finance pipeline quarantines negative amounts for review instead of silently publishing (validity).
Guidelines

Best practices

  • Start with corn-row smoke checks (count, grain, PK uniqueness, freshness) before fancy ones.
  • Assert at the boundary you own: check outputs, and check the transformation, not the BI tool.
  • Make gates explicit in code with owners and severity; document false-positive escapes.
  • Err on the side of blocking publish: silent bad data costs more than a paged engineer.
Watch out

Common mistakes

  • Writing 40 checks nobody reads and then ignoring failures — gates become noise.
  • Gating at the dashboard instead of the pipeline, so consumers see 'pipeline green, numbers wrong'.
  • Hard-coding expected counts that drift with the business — refresh thresholds or use ratios.
  • Testing only the head rows, missing volume-scale skew that kills performance.
Root cause analysis

Common issues

Error
Gate fires but engineering disagrees the data is wrong
Root cause

Check was written against a stale expectation of the data

Fix

Re-baseline the assertion with data owners, not by disabling it silently.

Error
Duplicate orders flood a daily table after a retry
Root cause

Extraction re-ran the window with an append strategy

Fix

Make the load idempotent (upsert/full-refresh) and enforce PK uniqueness in the gate.

Error
SLA says on-time every day yet customers complain of stale dashboards
Root cause

Freshness measured at load, not at consumer-visible model update time

Fix

Measure the gate on the served model's latest timestamp, matching consumer experience.

When things break

Debugging tips

  • A failing uniqueness check? Find the first duplicate and its source; usually a join or re-runnable extraction.
  • A freshness breach? Timebox: source lag, extract lag, transform lag, gate lag.
  • Compare a check's historical trend before assuming it's a bug — seasonality is real.
  • Reproduce wrong values with the smallest set: one key, one window, one column.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Checking that every order ID appears exactly once exercises which dimension?

2

A quality gate should notify after data is already served to consumers.

Go deeper

Official documentation

Great Expectations docs
docs
dbt tests
docs