Data Orchestration Best Practices
The same five habits, whichever orchestrator
Orchestration reliability is the same few habits: small idempotent tasks, explicit dependencies, retries with backoff, quality gates between stages, and alerting that means something. Apply them in Airflow, Dagster, or Prefect.
What you will learn
- Apply retry/backoff policies that heal while escalating
- Insert quality gates between stages, not just at the end
- Alert on outcomes, not on events
What is this?
Orchestration done well looks boring: tasks short and idempotent, dependencies explicit, retries bounded with backoff, a quality gate after each meaningful stage, and alerts that carry an owner and a hypothesis. Whatever the tool, the discipline — not the technology — creates the reliability.
Why do you need it?
- Most data outages are orchestration-shaped, and most are prevented by these habits.
- The same guidance transfers between Airflow, Dagster, and Prefect, so it's the durable skill.
- Quality gates turn 'is it done?' into 'is it correct?' at every stage.
Mental model
A relay race with checkpoints and a coach's traffic light
Orchestration with gates
Baton passes (dependencies) are checked; a drop (task failure) retries within a lap (backoff) and only then pages the coach (on-call). Checkpoints verify lap times (quality) not just that runners finished.
How it works
- Retries are bounded: 2-3 retries, exponential backoff, then alert — heal quick failures, page real ones.
- Idempotency first: any node reruns safely, which makes retries and backfills non-issues.
- Quality gates: after landing, after modeling — assert row counts, grains, freshness before proceeding.
- Alerting by outcome: page only where data correctness/freshness risk is real; aggregate the noise.
- Runbooks per DAG: owner, failure playbook, and remediation steps written before the incident.
Retry → gate → alert ladder.
Examples
Backoff + gate + alert in config
task_defaults: retries: 3 retry_delay: 300 # seconds retry_delay_factor: 2 # exponential backoff max_retry_delay: 1800 stages: - land (gate: rows_landed > 0) - model (gate: no dup grain key) - publish (gate: freshness < 4h)alert: on_failure: page #data-oncall escalate_after: "3 consecutive failures"Retry policy heals; gates prove; alerting only breaks through to humans when it means something.
In production
- A nightly DAG retries a flaky warehouse query twice then pages — no false alarms, real failures paged.
- A mart build gates on 'no dup order_line_key' before publishing, catching upstream drift in CI-style fashion.
- An ops runbook for 'gaps in raw landing' is written and rehearsed before the first real gap.
Best practices
- Bias gates early: a landing gate is 10x cheaper than a publish gate.
- Make alerts self-describing: owner + hypothesis + fix link, not just 'task failed'.
- Review retry policies quarterly; retry-forever settings breed chronic ignored failures.
- Run pipelines on data intervals (not wall-clock) so replays and backfills are the same code path.
Common mistakes
- Infinite retries, so a permanent schema bug quietly retries for a week.
- Alerting on every task event, drowning the on-call in noise until real pages are ignored.
- No quality gates: the pipeline 'runs' while silently emitting wrong numbers.
Debugging tips
- A 'failed then succeeded on retry' honestly means flaky: log which phase and error class for the review.
- When alerts are ignored for weeks, the problem is policy, not the pipeline — reduce noise at the source.
- Gate failures are the cheapest bugs; triage previous gates' failure types to find upstream rot.
Interview checkpoint
Hands-on lab
- Model retries with exponential backoff in code
- Insert a quality gate between stages
- Escalate only after N consecutive failures
Retry + Gate + Escalate in a Toy Pipeline
35 min · intermediate
Quick quiz
A landing gate asserting 'rows_landed > 0' prevents…
Infinite retries ensure a broken pipeline eventually succeeds.