Pipeline Orchestration
DAGs, dependencies and retries — the nervous system
Orchestration is the scheduler and dependency manager of ML pipelines: what runs, in what order, on retry, and with what compute. It's the difference between a script and an operation.
What you will learn
- Model a training workflow as a DAG
- Configure retries, dependencies and compute per step
- Recover from failures without replaying the world
What is this?
Orchestration runs multi-step ML workflows on a schedule with explicit dependencies: a DAG that says 'features before training before evaluation'. Orchestrators (Airflow, Kubeflow Pipelines, Argo) handle ordering, retries, backfills, compute allocation and logging across steps.
Why do you need it?
- Without a DAG, 'order' is hopeful sequencing in code — brittle and unrecoverable.
- Retries and backfills become first-class operations instead of manual re-runs.
- Long training jobs need paging, compute gating and checkpoint-aware recovery.
- Multiple people/teams share one definition of how work actually flows.
Mental model
An airport's departure board with gates and hold times
Pipeline orchestration
Departures (steps) have dependencies — fuel before board, board before pushback. A central board (DAG) sequences them, honest about delays and retries, so a stuck flight never blocks boarding of an unrelated one.
How it works
- You declare steps and their dependencies as a DAG (Python code or YAML).
- The scheduler evaluates triggers and starts ready steps in order.
- Each step runs in its designated compute (container, K8s, batch) with resource limits.
- Retries with backoff recover transient failures; state is persisted after each step.
- Logs, artifacts and run provenance are captured for the pipeline run.
- Failures page and leave the DAG in a known state for resume or backfill.
A training DAG and its dependency edges.
In production
- Airflow runs a nightly data → features → train → register DAG with per-step retries.
- Kubeflow Pipelines schedules training on K8s where the train step claims a GPU.
- A sensor step waits for a new data version before triggering the DAG.
Best practices
- Keep steps hermetic: each has explicit inputs/outputs and no hidden side effects.
- Set per-step retries with sensible backoff; never infinite loops.
- Enable checkpointing on long training so a retry skips completed iterations.
- Version the DAG definition in git; runs reference the definition version.
Common mistakes
- Steps that 'work on the current directory' — shared global state makes DAG unreplayable.
- Dependency expressed in code (sleep 3600) instead of a DAG edge.
- Retried steps that double-write artifacts because they weren't idempotent.
- No resource limits — a fat training step starves the whole cluster.
Debugging tips
- DAG stuck? Check the waiting step's unmet dependency or sensor.
- Retried but still failing? Read the step's container logs for the real error, not the retry summary.
- Backfill window off? Confirm the DAG's data partitioning matches the run dates.
Interview checkpoint
Hands-on lab
- Model a training workflow as explicit steps with dependencies
- Retry a failing step with backoff
- Checkpoint per-step state so recovery skips completed work
Run a Deterministic Pipeline
30 min · intermediate
Quick quiz
Which property matters most for orchestrator recovery?
Orchestration retries should recover transient failures with bounded backoff.