Python for Data Engineering
Intermediate 15 min read Python for Data Engineering

Exception Handling

Design your failure modes on purpose

Raise early, catch late, and never swallow errors — production pipelines are as much about deciding what must fail as about writing code that succeeds.

exceptionstryexceptfail loudretrieslogging
Prerequisites: Functions and Modules
Before we start

What you will learn

  • Write try/except/else/finally without swallowing errors
  • Distinguish recoverable errors (retry) from permanent ones (raise)
  • Chain and log exceptions with context for on-call debugging
Introduction

What is this?

Exception handling in data engineering is a policy decision: which failures are recoverable, which must abort, and what evidence each failure leaves behind. Done well, a crashed pipeline tells you exactly why it crashed; done poorly, it 'succeeds' with silent data loss.

Motivation

Why do you need it?

  • A bare `except:` that swallows errors turns a broken extraction into a believable empty table.
  • Retries are only safe where idempotency guarantees a retry can't double-write.
  • Logged context (what file, which partition, which row) is what makes a 3am page actionable.
Think about it

Mental model

Imagine

A smoke detector that stays silent

Think of

A swallowed exception

Silence implies safety. If your code catches every failure and keeps going, nobody knows the house is burning — until the data is gone.

Mechanics

How it works

  • `try` guards the risky operation; `except` handles the specific exception types you can actually recover from.
  • `else` runs only on success; `finally` always runs for cleanup (closing connections, releasing locks).
  • `raise ... from err` chains the original exception, preserving the full traceback.
  • Retry wrappers distinguish transient errors (ConnectionError, TimeoutError) from permanent ones (ValueError on bad data).
  • An outer boundary logs the final failure with structured context and re-raises so orchestrators see it.

A resilient extract-with-retry boundary.

Extract attempt
Transient error?
Permanent error?
Success
Log context
Hands-on

Examples

A retry-and-fail-loud extractor

A retry-and-fail-loud extractor
import time, logging
log = logging.getLogger(__name__)
 
def fetch_with_retry(make_request, attempts=3):
    for n in range(attempts):
        try:
            return make_request()
        except (ConnectionError, TimeoutError) as e:
            if n == attempts - 1:
                raise RuntimeError("extract failed permanently") from e
            time.sleep(2 ** n)
    raise RuntimeError("unreachable")

Only transient errors retry; the last failure re-raises a clear root cause that orchestrators see.

Real world

In production

  • A Kafka producer retries on retriable send errors but fails the DAG immediately on serialization errors.
  • A file loader validates row count and raises if output is empty — empty-but-green is the most dangerous silent failure.
  • A quality gate catches assertion failures, logs offending partitions, and aborts the publish step.
Guidelines

Best practices

  • Catch the specific exceptions you can handle; never a bare `except:` in long-running pipelines.
  • Fail loud at boundaries: extraction, validation, and publish each must raise on any anomaly.
  • Attach context (partition, source, row_id) to the exception message or logged fields.
  • Retry only idempotent work, with exponential backoff and a max attempt count.
Watch out

Common mistakes

  • `try: ... except: pass` hiding the error and shipping empty data.
  • Retrying permanent failures forever, burning API quota and delaying the alert.
  • Catching an exception and re-using the variable, dropping the traceback.
  • Not wrapping a retry loop in a timeout, so a hung API call stalls the whole DAG.
Root cause analysis

Common issues

Error
Pipeline green but table empty
Root cause

A swallowed exception made the run complete with zero output

Fix

Add row-count validation that raises on empty output, and replace bare except with specific handlers.

Error
Retry storm hits the upstream API
Root cause

Retrying permanent errors (400s) without distinguishing status codes

Fix

Retry 5xx and timeouts only; treat 4xx as permanent and fail fast.

When things break

Debugging tips

  • Read the full traceback chain (`from ... import err`) — the root cause is almost always the deepest frame.
  • Add run_id/partition fields to every log line so a failed run can be replayed exactly.
  • Temporarily set `PYTHONUNBUFFERED=1` to see logs live while debugging a stuck worker.
Interviews

Interview checkpoint

Practice

Hands-on lab

Bulletproof an Extraction with Retries and Gates
25 min · intermediate
In progress
Objectives
  • Implement a retry wrapper for transient failures
  • Add a fail-fast empty-check after extraction
  • Prove permanent failures are visible, not swallowed

Bulletproof an Extraction with Retries and Gates

25 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which pattern is the correct production default for an unexpected exception?

2

Retrying a non-idempotent operation is safe if you retry quickly.

Go deeper

Official documentation

Errors and Exceptions — Python docs
docs
PEP 654 — Exception groups
docs