ETL / ELT
Intermediate 14 min read ETL / ELT

Airflow in Practice

The scheduler is a dependency graph, not a cron

Airflow schedules and orchestrates tasks as DAGs. Making it reliable in production means: small idempotent tasks, dynamic-DAG discipline, backfill parameters, proper connections/secrets, and ruthless DAG hygiene.

AirflowschedulerDAG filebackfillconn_idretry
Prerequisites: DAG Design
Before we start

What you will learn

  • Write a DAG file that is deterministic and dynamic-safe
  • Configure retries and backoff so failures are cheap
  • Manage secrets with Connections, never plaintext
Introduction

What is this?

Airflow reads DAG definitions from Python files, the scheduler builds task instances for each schedule, and a worker fleet executes them. Production-grade Airflow is about hygiene: DAGs that parse fast and deterministically, secrets through Connections, backfills via parameters, and alerts on retry exhaustion.

Motivation

Why do you need it?

  • It is the default orchestrator most data teams meet first.
  • Most Airflow pain is the same five habits, all preventable.
  • The patterns here (idempotency, backfill, secrets) generalize to Dagster/Prefect.
Think about it

Mental model

Imagine

A station dispatcher with a printed timetable

Think of

Airflow scheduler + DAG

The timetable (DAG file) says train K leaves after train J arrives. The dispatcher (scheduler) re-checks each departure window. Deviation (a delayed train) is handled by retry/backoff, and cancellations (skip) don't wreck the rest of the line.

Mechanics

How it works

  • DAG files live in dags/; the scheduler parses them (also DAG folders must stay light).
  • Each schedule_interval triggers a DagRun; tasks become TaskInstances executed by workers via the executor.
  • Retries count and retry_delay apply per task; failing tasks mark downstream skipped until resolved.
  • Backfill runs the DAG for a past interval, honoring dependencies and idempotency.
  • Connections (conn_id) hold credentials in the metadata DB or a secrets backend — never in code.

Scheduler → executor → workers for one DAG run.

DAG file
Scheduler
Task queue
Workers
XCom / callbacks
Reference

Key commands

airflow dags backfill shop_daily --start-date 2026-01-01 --end-date 2026-01-05

Re-run a historical interval using the same DAG, honoring dependencies.

airflow tasks clear shop_daily --task-regex model_orders --start-date 2026-03-14

Reset only the failing subtree for replay.

airflow connections add db --conn-type postgres --conn-host ... --conn-login ...

Store credentials in Connections, never in DAG code.

Hands-on

Examples

A production-shaped DAG file

A production-shaped DAG file
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash import BashOperator
 
default_args = {"retries": 2, "retry_delay": timedelta(minutes=5),
                "owner": "data-team"}
with DAG("shop_daily", start_date=datetime(2025,1,1),
         schedule_interval="@daily", catchup=False,
         default_args=default_args, tags=["finance"]) as dag:
 
    raw = BashOperator(task_id="land_raw",
        bash_command="python /ops/ingest.py --ds {{ ds }}")
    build = BashOperator(task_id="build_mart",
        bash_command="dbt run --select mart_sales --vars '{day: {{ ds }}}'")
    raw >> build

Deterministic, idempotent tasks; ds templating gives replay without new code.

Real world

In production

  • A 20-table warehouse loads as one DAG with per-domain tasks and a final freshness gate.
  • A backfill on a historical interval replays the same graph with catchup-disabled logic.
  • Retry alerts page on the third failure, not the first, thanks to task-level retry config.
Guidelines

Best practices

  • Keep DAG files thin: import nothing heavy, compute nothing at parse time.
  • Use catchup=False for new DAGs, then deliberate backfills.
  • Constrain retries (2-3) and backoff so transient errors heal, chronic ones page.
  • Store every credential in Connections or a secrets backend.
Watch out

Common mistakes

  • Loading entire libraries at import time, making scheduler parses crawl.
  • Secrets in DAG files and git history — a compliance horror story.
  • catchup=True surprises, silently replaying a year of intervals on first deploy.
When things break

Debugging tips

  • An un-scheduled DAG is usually a parse error; check the scheduler log for 'failed to parse'.
  • A stuck task is often a worker-less executor or a leaked running process, not logic.
  • Use airflow dags list-import-errors to see DAG files that broke parsing.
Interviews

Interview checkpoint

Practice

Hands-on lab

Author, Validate, and Replay an Airflow-Style DAG
35 min · intermediate
In progress
Objectives
  • Write a real DAG file with retries and ds templating
  • Statically validate task dependencies
  • Simulate a backfill over historical dates with idempotent outputs

Author, Validate, and Replay an Airflow-Style DAG

35 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Databricks '{{ ds }}' templating exists in Airflow so that…

2

Secrets belong in Connections or a secrets backend, never inline in DAG code.

Go deeper

Official documentation

Airflow docs
docs
Airflow dynamics & templating
docs
Connections & secrets
docs