Data Engineering Foundations
Beginner 14 min read Data Engineering Foundations

The Data Lifecycle

Where problems actually live along the way

Data moves through a predictable lifecycle — capture, ingest, store, transform, serve, maintain. Knowing which phase you are in tells you which problems matter.

data lifecycleingesttransformserveretainpurge
Prerequisites: What is Data Engineering?
Before we start

What you will learn

  • Map any pipeline onto the six lifecycle phases
  • Explain which quality issues dominate each phase
  • Apply lifecycle thinking to define freshness and retention requirements
Introduction

What is this?

The data lifecycle describes the journey of a piece of data: it is captured at a source, ingested into your systems, stored, transformed into useful shapes, served to consumers and eventually retained or purged. Every phase has its own failure modes, and most pipeline bugs are 'right phase, wrong expectations'.

Motivation

Why do you need it?

  • It gives a shared vocabulary: teams can say 'we have an ingestion backlog' instead of 'data is broken'.
  • It surfaces the right SLAs: freshness is defined per phase, not globally.
  • Retention and purge decisions are lifecycle decisions, and they are legal obligations (GDPR, SOC2).
  • Root-causing is faster because you enumerate which stage could have introduced a defect.
Think about it

Mental model

Imagine

A food supply chain

Think of

The data lifecycle

Produce is harvested (capture), shipped (ingest), stored (store), prepared (transform), served (serve) and expires (retain/purge). Each stage has freshness rules and spoilage checks — data is the same.

Mechanics

How it works

  • Capture: an event is logged, a row is written, or a file lands in a bucket.
  • Ingestion: an extractor copies data from the source into your environment (pull/push, batch/stream).
  • Storage: raw files land in a lake or staging tables in a warehouse.
  • Transformation: SQL or code turns raw data into modeled tables (clean, dedupe, aggregate).
  • Serving: consumers query final tables via BI tools, notebooks or APIs.
  • Maintenance: retention policies, archival and deletion are executed on schedule.

The six data lifecycle phases.

Capture
Ingest
Store
Transform
Serve
Maintain
Real world

In production

  • An analytics team defines freshness as 'the transform phase must finish by 6:00am'; ingestion lag is monitored separately from modeling lag.
  • A healthcare company archives 7 years of PHI to cold storage and purges after that, driven by a lifecycle retention policy.
  • A gaming company separates capture (client telemetry), ingest (Kafka), store (S3), transform (Spark) and serve (Redshift) so each team owns one stage and one SLA.
Guidelines

Best practices

  • Write down retention windows and purge logic before storing data, not after.
  • Tag every dataset with capture time and expected freshness at each phase.
  • Instrument each phase boundary with counts so you can say exactly where data stops advancing.
  • Define, per phase, who owns the SLA and who gets paged.
Watch out

Common mistakes

  • Engineering only the transform phase and ignoring stale ingestion — the freshest transform in the world can't fix a 3-day-old source.
  • Storing everything forever 'just in case' — cost and compliance both blow up.
  • Mixing timezones across phases so 'midnight' means three different things.
  • No phase boundary metrics, so a pipeline break is discovered by a customer.
Root cause analysis

Common issues

Error
Dashboard is fresh but shows yesterday's data
Root cause

A transform completes on time, but ingestion for one source is silently delayed

Fix

Add per-source expected arrival monitors in the ingest phase, not the serving phase.

Error
Storage costs doubled with no new pipeline
Root cause

No retention policy; duplicates and archives accumulate

Fix

Classify datasets by lifecycle stage and apply tiering/purge rules.

Error
Backfill overwrote newer data
Root cause

Re-running an old load without protecting the latest partition

Fix

Add partition guards and make backfills explicit about which window they touch.

When things break

Debugging tips

  • Check phase counters in order: if capture is fine but ingest count is flat, the extractor is stuck.
  • Compare row counts between phases to find the exact hop where data is lost.
  • Look at timestamps (event time vs processing time) to distinguish capture lag from pipeline lag.
  • Keep a raw copy untouched by transforms; it is your source-of-truth for replay.
Interviews

Interview checkpoint

Practice

Hands-on lab

Walk the Data Lifecycle with SQLite
25 min · beginner
In progress
Objectives
  • Simulate capture, ingest, store, transform and serve with a tiny SQLite database
  • Run a transformation that cleans raw rows into an analytical shape
  • Prove the pipeline is idempotent by running it twice safely

Walk the Data Lifecycle with SQLite

25 min · beginner

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

In which phase does a 'stale source' problem first manifest?

2

Retention policies should be designed after data has been in production for a while.

Go deeper

Official documentation

Data lifecycle management — AWS
docs