Python for Data Engineering
Beginner 14 min read Python for Data Engineering

Python Basics Review

The 20% of Python you use 80% of the time

A fast, honest refresher of Python syntax, typing, control flow and built-ins — the subset that actually shows up in data pipelines.

pythonsyntaxtypingbuiltinspipelines
Prerequisites: The Python Data Ecosystem
Before we start

What you will learn

  • Read and write idiomatic Python for data tasks
  • Use built-ins and comprehensions instead of hand-rolled loops
  • Deal with mutable defaults, scoping and the Python data model correctly
Introduction

What is this?

This is the working subset of Python you will actually write as a data engineer: sequences and mappings, comprehensions, slicing, unpacking, string formatting, `with` blocks, and the collection of syntactic traps that produce subtle pipeline bugs.

Motivation

Why do you need it?

  • Pipelines are full of small data manipulations — filtering lists, building dicts, parsing strings — where built-ins are faster and clearer than loops.
  • Comprehensions and generator expressions become the core of later vectorization work.
  • Knowing the traps (mutable defaults, aliasing, `is` vs `==`) prevents the classic silent bugs in ETL code.
Think about it

Mental model

Imagine

Learning the keyboard shortcuts of a text editor

Think of

Learning Python built-ins

You can do everything with arrow keys and menus, but the idiomatic shortcuts are faster and fewer accidental edits. Same with Python: `for` loops work, comprehensions are the shortcut.

Mechanics

How it works

  • `list`, `dict`, `set`, `tuple` are the workhorse containers, each with its own identity/copy semantics.
  • Comprehensions build a new container from an iterable in one expression: `[x * 2 for x in xs if x > 0]`.
  • Slicing `l[start:stop:step]` returns copies for lists (careful: it builds a new list).
  • `with open(...)` guarantees file handles are closed even on exceptions.
  • `is` compares identity, `==` compares value — a frequent source of subtle bugs when comparing singletons or interned strings.

The core building blocks of a Python data script.

Read input
Transform with built-ins
Aggregate
Serialize output
Done
Hands-on

Examples

Idiomatic data line processing

Idiomatic data line processing
lines = ["a:1", "b:2", "a:3"]
from collections import Counter
totals = Counter()
for line in lines:
    k, v = line.split(":")
    totals[k] += int(v)
print(dict(totals))  # {'a': 4, 'b': 2}

Unpacking, the Counter container and `with` semantics are the idioms you will repeat in every parser you write.

Real world

In production

  • A parse step reads a TSV, filters malformed rows with a comprehension, and aggregates counts with Counter before writing output.
  • A config loader uses dict unpacking (`{**base, **overrides}`) to merge environment overrides safely.
  • An extractor walks S3 keys with sorted(glob) and slicing to process the latest N partitions deterministically.
Guidelines

Best practices

  • Prefer built-ins and comprehensions over explicit loops when building new containers.
  • Use `pathlib.Path` for all file handling — it prevents the slash/escape bugs Windows users hit.
  • Write functions that take and return immutable containers so transformations are easy to test.
  • Match the project's Python version in a `.python-version` or pyproject to avoid drift.
Watch out

Common mistakes

  • Default argument `def f(xs=[])` — the mutable default is shared across calls and silently accumulates.
  • Using `==` to compare with `None` or checking `if x == True` instead of truthiness.
  • Mutating a dict while iterating over it (`RuntimeError: dictionary changed size during iteration`).
  • Slicing lists in hot loops, creating copies and wrecking memory use on large inputs.
Root cause analysis

Common issues

Error
dict changed size during iteration
Root cause

Mutating the dict you are looping over

Fix

Iterate over `list(d.items())` or build a new dict with a comprehension.

Error
Function behaves differently on second call
Root cause

Mutable default argument reused across calls

Fix

Use `def f(xs=None): xs = xs or []`.

When things break

Debugging tips

  • Use a linter (ruff) nightly — it catches mutable defaults, accidental shadowing and undefined names before runtime.
  • `python3 -c` one-liners are your REPL: test a snippet without creating files.
  • Set `PYTHONWARNINGS=error` once locally to surface deprecations early.
Interviews

Interview checkpoint

Practice

Hands-on lab

Python Boot-Camp: Files to Dict to Summary
25 min · beginner
In progress
Objectives
  • Read a small CSV with pathlib and the csv module
  • Transform rows with comprehensions and unpacking
  • Aggregate with Counter and write a summary file

Python Boot-Camp: Files to Dict to Summary

25 min · beginner

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

What is the result of `[x * 2 for x in range(3) if x != 1]`?

2

`with open(path) as f:` guarantees the file closes even if an exception is raised inside the block.

Go deeper

Official documentation

Built-in Types — Python docs
docs
pathlib — Python docs
docs