Reliability Engineering for Data

• Engineer failure-safety: idempotency, retry, backoff, circuit breakers
• Load-test and game-day pipelines against realistic failure modes
• Run reliability as a budgeted practice with burn-driven priorities

• Data pipelines fail in slow, partial ways (silent drops, skipped feeds) that a crash-detector never sees.
• Idempotency is the cheapest superpower: a re-run is a safe action, so recovery = rerun.
• Load and chaos games reveal the failure modes your monitoring didn't predict.
• Error budgets convert 'be more careful' into 'spend budget on the top burners' — quantified prioritization.

You don't hope engines never fail — you certify the plane keeps flying (and lands) on one. Pipelines get the same: retries, fallbacks and recovery paths designed in before the failure happens, then proven by drills.

• Idempotency: every step overwrites/upserts by key; re-runs converge to the same state.
• Retry/backoff: transient dependency errors retried with exponential backoff + jitter; per-request timeouts.
• Circuit breaker: on persistent source outage, fail fast to a fallback (last-good feed) rather than hammering.
• Recovery: checkpointing and replay from the last commit; rebuild from lineage when state is lost.
• Load testing: simulate a worst-week volume; verify batch windows and stream lag stay within SLO.
• Chaos games: kill a broker, kill a backfill, inject a silent drop — measure recovery and MTTR improvements.

• A warehouse pipeline is idempotent by design: after a mid-run kill, on-call re-runs it and the tables converge — no manual cleanup.
• A streaming product run circuit-breakers a degraded source for 10 minutes, serving last-good data, and auto-heals when the feed recovers.
• A platform runs a quarterly chaos game — kill the nightly scheduler — and the team's MTTR drops each quarter because the drills harden runbooks.

• Make idempotency a non-negotiable design rule at code review; it's the difference between 'rerun' and 'disaster'.
• Retry with jitter; retries without jitter become thundering herds during outages.
• Design the fallback path before the outage: what is 'last-good' data and how stale may it be?
• Game-day cheaply: fix one drill scenario per quarter and measure MTTR before/after.

• Retrying forever at fixed intervals and wedging the whole queue on a flaky dependency.
• Idempotency by fiction — appends and monotonic counters that break on replay.
• Load planning on best-case growth instead of worst-week; SLO breaks exactly at the 'surprise' month.
• Chaos games that run only the happy path, drowning in the scenarios they never rehearsed.

• A partial-run mess after a crash → check idempotency path: full-overwrite and key UPSERT vs appends.
• Retry storms → look at retry policy (no jitter, no backoff) at the dependency level, not just the code.
• Unexplained lag during a 'healthy' period → game-day the slow consumer: slow sink, hot partition, or 3rd-party slowdowns behave alike.
• Failover that 'worked yesterday' fails today → the fallback source drifted; validate the last-good copy freshness in the drill.