Data Engineering Foundations
Intermediate 15 min read Data Engineering Foundations

Data Pipeline Fundamentals

Steps, dependencies and the properties that make them trustworthy

A data pipeline is a DAG of steps that move and transform data. Reliable pipelines are observable, idempotent, retryable and monitored against SLAs.

data pipelineDAGidempotencyretryorchestrationSLA
Prerequisites: The Data Lifecycle
Before we start

What you will learn

  • Describe a pipeline as a DAG with dependencies and schedules
  • Apply the four reliability properties: idempotent, retryable, observable, SLA-aware
  • Diagnose failures that look like tool problems but are design problems
Introduction

What is this?

A data pipeline is a directed acyclic graph (DAG) of tasks that extract, transform and load data on a schedule or on event triggers. Reliability comes from pipeline design properties — idempotency, retryability, observability and explicit SLAs — far more than from any single tool.

Motivation

Why do you need it?

  • Pipelines fail: sources change, volumes spike, software is deployed — the design must absorb sloppiness.
  • Idempotency + retries means a failed 3am run can self-heal at 4am and still be correct.
  • Observability is what turns a silent data error into a discoverable, paged one.
  • An SLA turns 'the dashboard is late' into a measurable, owned promise with an owner.
Think about it

Mental model

Imagine

A tutorial quiz's grading pipeline

Think of

A data pipeline

Every time a student submits, the grader runs the same steps on the same inputs and produces the same score; if a grader crashes midway, re-running is safe because it overwrites rather than appends.

Mechanics

How it works

  • Tasks are nodes; dependencies are edges. A task with no unmet dependencies can run.
  • The orchestrator schedules tasks, retries failed tasks, and backfills missed windows.
  • Each task declares outputs (tables/files); consumers depend on those outputs, not on the tool.
  • Idempotent tasks replace their output wholesale or dedupe, so retries and re-runs are safe.
  • Monitoring compares expected vs actual runs, durations and row counts to surface SLA violations.

A pipeline DAG with a failed task blocked downstream.

Extract API A
Extract DB B
Join & clean
Load gold table
Notify consumers
Real world

In production

  • Airflow schedules a nightly DAG; if the incremental extract of the replica DB fails, the DAG retries three times then pages on-call, leaving the downstream model untouched since transforms only run on full success.
  • An event-driven pipeline triggers on a new batch file in S3, validates the schema, and loads each partition idempotently.
  • dbt DAGs model the warehouse; a failed test (duplicate key) fails the run before certified consumers see it.
Guidelines

Best practices

  • Make every task idempotent: full-refresh or upsert with a natural key.
  • Add retries with backoff and make retries safe (retry the whole window, not a fragment).
  • Record run metadata (start, end, rows_in, rows_out, version) for every task.
  • Alert on SLA not just failure: late-but-running still violates freshness.
Watch out

Common mistakes

  • Writing tasks that append instead of replace, causing duplicates on every retry.
  • Retrying indefinitely without backoff on a deterministic failure (wastes compute, delays diagnosis).
  • No row-count assertions, so pipelines 'succeed' while silently dropping a whole source.
  • Scheduling on wall-clock time instead of on data arrival, then racing the source's freshness.
Root cause analysis

Common issues

Error
Task 'succeeds' but the table is empty
Root cause

Filter or source-read bug landed as valid SQL; no assertions on rows

Fix

Add row-count and completeness checks that fail a run when rows are below threshold.

Error
DAG retries make duplicates worse
Root cause

Non-idempotent append task retried the same window

Fix

Make tasks full-refresh/upsert keyed by a natural key so retries overwrite.

Error
Dashboard late but no one paged
Root cause

Alerts watched task failures, not freshness SLAs

Fix

Monitor latest-refresh timestamp vs now-expected; alert when staleness breaches the SLA.

When things break

Debugging tips

  • Check run status + row counts together: succeeded-but-empty is the classic silent failure.
  • Re-run just the failed task and its dependents — don't re-run the whole DAG if you can avoid it.
  • Compare run logs to the schedule: a task that never ran on time points to orchestration.
  • Use the version/timestamp fields to prove exactly which code and source state produced an output.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which property lets you safely re-run a failed pipeline window?

2

A pipeline that always 'succeeds' but occasionally produces empty tables is reliable.

Go deeper

Official documentation

Apache Airflow — DAG best practices
docs