DAG Design
Dependencies made explicit, failures made cheap
A DAG turns dependencies into order. Good DAGs are small, idempotent, restartable, and clearly owned. Bad DAGs hide cycles, replay hell, and cross-team coupling inside one god graph.
What you will learn
- Model a pipeline as a DAG with real dependency edges
- Apply idempotency so any node reruns cleanly
- Design for restart-from-any-node and clear ownership
What is this?
A DAG (directed acyclic graph) encodes 'what depends on what' so an orchestrator runs nodes in a valid order, retries failures, and skips nothing. The discipline is design: keep DAGs small and focused, make every task idempotent, and let dependencies, not cleverness, decide the run.
Why do you need it?
- REQUIRED slug: de-foundations links here as the canonical orchestration reference.
- Most pipeline outages are dependency bugs wearing a 'transient' costume.
- Idempotency plus small DAGs is 80% of the entire orchestration skill.
Mental model
A recipe with steps and no order enforcement
A DAG
You can't glaze the roast before trussing it; a DAG stops you from trying. If you re-discover you forgot the glaze (rerun a task), everybody downstream still gets finished (idempotent replay).
How it works
- Nodes = tasks; edges = 'must finish before'. The orchestrator topologically orders and parallelizes free branches.
- Idempotency: re-running a task produces exactly the same output — delete-then-write, or hash-based writes.
- Retries and backoff live on edges/nodes; a failure marks only its subtree for retry.
- Clear ownership: one team per DAG, one on-call, one runbook.
- Branching (skipped upstream) and partial failures must not corrupt other branches.
A small DAG: extract → validate → two modeling branches.
Examples
A DAG, and why every node can rerun
with DAG("shop_daily",...): extract = PythonOperator(land_raw) # idempotent: rewrite partition validate = PythonOperator(check_row_counts) # fails fast before models model_ord = SQLOperator(dbt_run("orders")) model_cust = SQLOperator(dbt_run("customers")) publish = SQLOperator(verify_freshness) extract >> validate >> [model_ord, model_cust] >> publish # Replay: if model_ord fails at 3am, only its subtree reruns on fix,# because every node above is idempotent.Explicit edges + idempotent tasks = the only two things most runs ever need.
In production
- A daily DAG fans out from a raw landing to per-domain models and converges at a publish gate.
- A streaming-shaped batch DAG uses sensors to wait for a Kafka partition checkpoint before modeling.
- A backfill DAG runs the same graph over historical partitions without touching live outputs.
Best practices
- One concern per DAG; make cross-team data flows contracts, not shared DAGs.
- Every task idempotent; delete-reinsert instead of blind-append.
- Add explicit validation nodes so failures happen early and cheaply.
- Keep tasks small enough that a single failure retries in under an hour.
Common mistakes
- Building one god-DAG with 400 tasks and three teams' jobs inside it.
- Non-idempotent tasks that double rows on retry — the oldest replay bug.
- Dependencies implied by sleep() or 'after cron time' instead of explicit edges.
Debugging tips
- A 'random' downstream failure usually has an upstream edge the DAG never declared.
- Replay the failing node alone; if it corrupts, idempotency was the lie.
- Draw the DAG before deploying it: cycles and orphan branches jump out on paper.
Interview checkpoint
Hands-on lab
- Represent a pipeline as tasks + dependency edges
- Topologically order tasks and detect a cycle
- Re-run an idempotent node with zero state drift
Model a DAG, Detect a Cycle, Prove Idempotency
35 min · intermediate
Quick quiz
A DAG guarantees that steps run in dependency order but also defines which branches run in parallel.
The cheapest fix for 'replaying a failed node produced double rows' is…