Python for Data Engineering
Advanced 16 min read Python for Data Engineering

Performance and Profiling

Measure first, optimize second, prove third

Profile before you optimize. cProfile, memory profiling and careful hot-path choices turn a 'slow script' into a defensible engineering measurement.

profilingcProfilememorymultiprocessinghot pathbenchmarking
Prerequisites: pandas Essentials numpy Essentials
Before we start

What you will learn

  • Profile CPU with cProfile and identify hot functions
  • Measure memory per line to catch leaks and duplications
  • Right-size multiprocessing vs unchanged single-process code
Introduction

What is this?

Performance work in Python is evidence-driven: cProfile shows cumulative and per-call time; tracemalloc/memory_profiler attribute memory growth to lines; profiling tells you whether to vectorize, parallelize, or (usually) just stop copying data.

Motivation

Why do you need it?

  • Optimizing the wrong function is wasted work — profilers remove the guessing.
  • Memory is usually the real constraint in data work; a leaked copy of a big frame kills a worker.
  • Multiprocessing helps only if the work is CPU-bound and the data-sharing overhead is small.
Think about it

Mental model

Imagine

A doctor taking vitals before prescribing

Think of

Profiling before optimizing

You wouldn't treat a cough with surgery. The profile is the blood test — it names the actual organ (function) that is slow before you prescribe a fix.

Mechanics

How it works

  • `python -m cProfile -o out.prof run.py` samples call counts and cumulative times.
  • pstats/`snakeviz` loads the profile: total_tt and percall columns find the hot functions.
  • `import tracemalloc` snapshots memory allocation sites; memory_profiler annotates lines.
  • CPU-bound loops parallelize via multiprocessing (pickling cost matters); I/O-bound work benefits from asyncio/threads.
  • Reduce copies first: reuse buffers, avoid accidental reallocation in tight loops.

The profile-then-fix loop.

Baseline
Profile CPU
Profile memory
Fix the hot path
Re-measure
Hands-on

Examples

The minimal profiling recipe

The minimal profiling recipe
python3 -m cProfile -o run.prof pipeline.py
python3 -m pstats run.prof  # interactive: sort cumulative / stats 20

cProfile attributes time to functions; pstats sorts by cumulative CT. Only then do you touch code.

Real world

In production

  • A batch job was 'slow'; cProfile showed str parsing, not SQL, dominated — a 10-line change cut runtime 40x.
  • A nightly memory spike was traced with tracemalloc to an accidental slice copy repeated per row.
  • A clean CPU-bound transform moved to multiprocessing with chunked inputs and measured near-linear speedup.
Guidelines

Best practices

  • Always record a baseline before optimizing and re-run the identical benchmark after.
  • Profile on a representative input — 2 rows misleads, 2 million doesn't.
  • Verify bottlenecks with code inspection (view right); the profile reports, not decides.
  • Treat memory growth as a bug: leak-stable pipelines are the benchmark.
Watch out

Common mistakes

  • Micro-optimizing before profiling — the 50-line refactor hits a 2% hot spot.
  • Adding multiprocessing which then dominates in pickling/serialization overhead.
  • Benchmarking on `time` with cached files or warm caches and comparing apples to oranges.
  • Leaving performance 'goals' to vibes instead of a repeatable benchmark script.
Root cause analysis

Common issues

Error
Runtime identical after 'optimization'
Root cause

The profiled hot path wasn't the one touched

Fix

Re-profile after the change and compare pstats tables before tuning further.

Error
Multiprocessing SLOWER than single process
Root cause

Pickling the input/output per worker exceeds the compute saved

Fix

Chunk work so each worker's share is large relative to serialization, or use shared memory/process pools with initializer-loaded data.

When things break

Debugging tips

  • For a sudden slowdown, diff profiles before/after a deploy — the new frame names the culprit.
  • Watch RSS with resource.getrusage in a loop to spot monotonic growth (leak) vs spikes.
  • When I/O-bound, profile syscalls (strace -c) — the bottleneck may be a single syscall per row.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

What does cProfile tell you that timing the whole script does not?

2

The first step in optimizing a slow pipeline is rewriting it with faster libraries.

Go deeper

Official documentation

The Python Profilers — docs
docs
tracemalloc — docs
docs