Python for Data Engineering
Advanced 16 min read Python for Data Engineering

Advanced Python Patterns

Compose, wrap, and log like a platform, not a script

Decorators, context managers, the `logging` discipline and pipeline-shaped abstractions are the patterns that let one small codebase serve dozens of production runs.

decoratorscontext managersloggingconfigurationpipeline patterns
Prerequisites: Performance and Profiling
Before we start

What you will learn

  • Write decorators and context managers that wrap pipeline steps
  • Configure structured logging once and reuse it everywhere
  • Recognize when an abstraction pays for itself versus when it hides the code
Introduction

What is this?

Once your pipeline is functions and modules, the next level is composition patterns: decorators add cross-cutting behavior (retry, timing, metrics) to functions without rewriting them; context managers manage resources; and disciplined logging gives every run a searchable, structured record.

Motivation

Why do you need it?

  • A `@retry` decorator applied to any extractor removes a whole class of boilerplate and bug surface.
  • Context managers guarantee file/db/connection cleanup on every exit path.
  • Structured logs (`{"run_id":..., "rows":...}`) are what alerting and dashboards parse.
Think about it

Mental model

Imagine

A security checkpoint wrapper around a building

Think of

A decorator around a function

The building's purpose doesn't change, but every entry now passes the checkpoint — access control, timing, logging applied uniformly without rebuilding the building.

Mechanics

How it works

  • Decorators wrap a function: `@retry` rewrites the call to add the retry loop around the original logic.
  • Context managers (`with step("extract") as s:`) start/stop timing, metric and logging entries.
  • The `logging` module with a configured formatter logs JSON per event; never `print` in production code.
  • Pipeline-shaped abstractions (a Step class, a run() returning metadata) make DAGs introspectable.
  • Configuration via pydantic/zod-style validated env keeps secrets out of code and defaults explicit.

Cross-cutting concerns wrapped around a core function.

@retry
@timed
with step(...)
core function
structured log
Hands-on

Examples

A reusable timing+metrics decorator

A reusable timing+metrics decorator
import functools, logging, time
log = logging.getLogger("pipeline")
 
def timed(name):
    def deco(fn):
        @functools.wraps(fn)
        def wrapper(*a, **k):
            t0 = time.time()
            result = fn(*a, **k)
            log.info({"event": "step", "name": name, "ms": (time.time() - t0) * 1000})
            return result
        return wrapper
    return deco
 
@timed("extract-orders")
def extract():
    ...

One decorator adds uniform timing/telemetry to any step, keeping core functions pure.

Real world

In production

  • `@retry(max=3)` wraps API extractors; `@timed` wraps every pipeline step for dashboards.
  • A `with warehouse_connection(...)` context manager commits on success and rolls back on exception.
  • A run() composer returns run metadata (start, end, rows) that the orchestrator stores as run records.
Guidelines

Best practices

  • Keep decorators small and orthogonal: one concern per decorator (retry, time, log).
  • Use context managers for anything that must be cleaned up; never rely on 'crash cleans up'.
  • Log structured events with a stable event name; grep by that name, not free text.
  • Add the run_id/partition to every log record from a single contextvar — never thread it by hand.
Watch out

Common mistakes

  • Decorating stateful functions and reproducing identical retries that double effects.
  • Over-abstracting: a Step class hierarchy with 50 subclasses hides logic behind ceremony.
  • print()-based logging that breaks under multiprocessing buffering.
  • Catching and logging inside the decorator AND the function — duplicate events mislead alerting.
Root cause analysis

Common issues

Error
Metrics double-count every run
Root cause

Decorator AND the caller both log the same event

Fix

Log at exactly one layer; standardize the decorator as the only emitter for that metric.

Error
run_id stale across parallel runs
Root cause

Module-level variable shared between workers

Fix

Use contextvars (thread-safe) or pass run_id explicitly through function signatures.

When things break

Debugging tips

  • Use functools.wraps so the decorated function's name/docstring survive — tracebacks stay readable.
  • Log exceptions at the boundary once, with exc_info=True, and let it propagate to the orchestrator.
  • For contextvar-based run_id, verify it resets between runs or events leak across pipeline runs.
Interviews

Interview checkpoint

Practice

Hands-on lab

Add Retry, Timing and JSON Logging to a Pipeline
30 min · advanced
In progress
Objectives
  • Wrap a flaky extractor with a retry decorator
  • Add structured JSON logging with run context
  • Verify logs are machine-parseable and complete

Add Retry, Timing and JSON Logging to a Pipeline

30 min · advanced

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which concern is a good fit for a decorator?

2

Functools.wraps preserves a function's name and docstring after decoration.

Go deeper

Official documentation

Functional programming HOWTO — decorators
docs
logging — Python docs
docs