Python for Data Engineering
Intermediate 14 min read Python for Data Engineering

Python Data Structures

Choosing the right container is a design decision

Lists, dicts, sets, tuples and the collections module are the raw material of every transform. Choosing the right structure is a performance and correctness decision.

listdictsettuplecollectionshashtable
Prerequisites: Python Basics Review
Before we start

What you will learn

  • Pick the right container for the access pattern (lookup, membership, order, dedup)
  • Explain the cost of list lookup vs dict/set membership
  • Use collections.deque, defaultdict and Counter in pipelines
Introduction

What is this?

Python's core containers trade off guarantees: lists preserve order with O(n) membership, dicts and sets give O(1) keyed access and dedup, tuples are immutable bundles. The `collections` module adds specialized tools — `defaultdict`, `Counter`, `deque`, `OrderedDict` — that data code reaches for daily.

Motivation

Why do you need it?

  • `x in big_list` is O(n); `x in big_set` is O(1). On a million keys that is the difference between seconds and instant.
  • dicts are the natural storage for mapping keys to rows, aggregations, and lookups.
  • Understanding hashability tells you why dict keys must be hashable and why custom keys need `__hash__`.
Think about it

Mental model

Imagine

A filing cabinet section vs a card index vs a phone book

Think of

list vs set vs dict

A list is papers in storage order (slow to search), a set is a de-dup bucket (only membership matters), a dict is a phone book (key → entry). Pick the tool that matches how you'll ask questions.

Mechanics

How it works

  • lists are contiguous arrays: fast append/pop at the end, O(n) insert/membership.
  • dicts store key→value using a hash table: O(1) average lookup and insert, order-preserving since 3.7.
  • sets are dicts without values: perfect for dedup and membership.
  • tuples are immutable lists — cheap, hashable, and safe to use as dict keys.
  • Counter is a dict subclass counting hashable items; defaultdict supplies missing keys with a factory.

Mapping containers to access patterns.

Ordered, repeated, index access
Lookup by key / aggregate
Membership + dedup
Immutable bundles / keys
Count / group / FIFO
Hands-on

Examples

Data-shaped dict patterns

Data-shaped dict patterns
from collections import defaultdict, Counter
rows = [("EU", "a"), ("US", "b"), ("EU", "c")]
by_region = defaultdict(list)
for region, sku in rows:
    by_region[region].append(sku)
counts = Counter(region for region, _ in rows)
print(by_region["EU"], counts)  # ['a', 'c'] Counter({'EU': 2, 'US': 1})

defaultdict avoids the get/set boilerplate; Counter collapses a whole aggregation to one line.

Real world

In production

  • A dedupe step uses a set of primary keys to cheaply reject already-seen records in a stream.
  • A fact-builder groups events into defaultdict(list) by natural key before emitting one row per key.
  • An orchestrator stores run state in a dict keyed by (dag_id, run_id) with tuple keys for atomicity.
Guidelines

Best practices

  • Use membership on sets/dicts, never lists, for hot-path contains-checks.
  • Prefer immutable tuples for record-packing so hashing and comparison are unambiguous.
  • Use Counter and defaultdict for aggregations instead of hand-rolled if/else.
  • Copy with `.copy()/copy.deepcopy` explicitly — assignment aliases, it does not copy.
Watch out

Common mistakes

  • Using a list where a set belongs, then discovering O(n) membership at a million rows.
  • Dict keyed by a mutable list — TypeError: unhashable type.
  • Aliasing a nested structure (`b = a['x']` then mutating b mutates a's data).
  • Iterating a dict and mutating it mid-loop.
Root cause analysis

Common issues

Error
TypeError: unhashable type: 'list'
Root cause

A dict key or set element is a mutable list

Fix

Convert to tuple (or frozenset for nested sets) before using as a key.

Error
The same data doubles in memory
Root cause

Aliased nested lists get mutated in place across two 'copies'

Fix

Use copy.deepcopy for nested structures, or immutable data (tuples).

When things break

Debugging tips

  • Surprise growth? Check if you're aliasing shared lists instead of copying.
  • KeyError when you 'know' the key exists? Check normalization (case/whitespace/base type).
  • O(n) slow-down on big inputs? Search for `in <list>` inside a loop and convert to a set.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

You must reject duplicate event ids while streaming a million events. Which structure speeds up membership checks?

2

A dict's key must be hashable, so mutable lists can never be dict keys.

Go deeper

Official documentation

collections — Python docs
docs
Data Structures — Python tutorial
docs