Comprehensions and Built-ins
Expressive, fast collection building
List/dict/set comprehensions build collections in one expression, and zip, enumerate, sorted, map and lambda compose the functional-clean style pandas users write every day.
What you will learn
- Rewrite loops as comprehensions
- Use generators for streaming memory
- Compose zip/enumerate/sorted/lambda idiomatically
What is this?
A comprehension builds a new collection by transforming or filtering each element: `[f(x) for x in seq if cond]`. Dict and set comprehensions extend it to mappings and memberships. Generators (parens or yield) build lazily, one element at a time. The companion built-in functions — zip, enumerate, sorted, min/max/sum — let you express whole transformations in a single line.
Why do you need it?
- Comprehensions are faster than append loops (dedicated bytecode) and far more readable.
- Generators keep 10M-row streams from materializing 10M-element lists.
- Clean, single-expression code is the fastest path to review-approval and debugging.
- They're the style pandas code is written in, so reading notebooks gets easier.
Mental model
An assembly line with an inspection filter
Comprehension pipeline
Items enter, a worker transforms each, a checker drops bad ones, finished products exit — `[transform(x) for x in items if ok(x)]` is that conveyor in one line.
How it works
- `[e for e in seq]` builds a list; `{k: v for ...}` builds a dict; `{e for e in seq}` builds a set.
- Order of clauses matches nesting: `[f(y) for x in a for y in x]` = nested for loops.
- A generator `(e for e in seq)` computes on demand — one item in memory at a time.
- zip pairs two iterables; enumerate adds indices; sorted(reverse=..., key=...) orders by a function.
- lambda creates small anonymous functions used as key= or filter functions.
Anatomy of a comprehension.
Examples
The daily comprehension toolbox
nums = [5, 12, 7, 19, 4]squares = [n ** 2 for n in nums]big = [n for n in nums if n > 8]pairs = {n: n**2 for n in nums}idx = {k: v for v, k in enumerate(nums)} # value → index for name, val in zip(["gdp", "pop"], [1000, 50]): print(name, val) top = sorted(nums, reverse=True)[:2]print(squares, big, idx[7], top)Every data prep step — transformations, filters, lookups, index mapping, ordering — fits in one line here. The dict comprehension inverted into an index is a frequent practical trick.
In production
- A feature builder converts a raw list of dicts into normalized feature dicts with one comprehension.
- A memory-constrained ingest streams a 20M-line file through a generator, bucketing counts without holding all lines.
- Validation code maps column names to dtype checks via a single dict comprehension over a schema.
Best practices
- Keep comprehensions to one for + one if; beyond that define a helper or loop.
- Prefer generator expressions for iterating big data single-pass.
- Write the plain loop first when the logic is subtle — then compress it.
Common mistakes
- Nested comprehensions (for a in for b in ...) becoming unreadable two-dimensional loops.
- Forgetting that a generator is single-use — it's exhausted after one pass.
- Using a comprehension for side effects; comprehensions should build values.
Debugging tips
- Decompose a failing comprehension back into an explicit for loop to find the throwing element.
- List(generator) materializes it temporarily for when you must see values.
- If a comprehension is slow, the filter/transform per element is the cost, not the comprehension itself.
Interview checkpoint
Hands-on lab
- Rewrite loops as comprehensions
- Aggregate with generator expressions
- Check the memory win of generators
Comprehensions at Scale
25 min · beginner
Quick quiz
`[x**2 for x in range(5) if x % 2 == 0]` produces:
A generator expression produces all values eagerly in memory.