Python for Analytics
Intermediate 13 min read Python for Analytics

Comprehensions and Generators

Expressive loops that stay memory-friendly

Comprehensions build collections from loops in one readable line; generators do it lazily, so a million-row transform never loads into memory at once.

list comprehensiondict comprehensiongeneratorlazymemoryiteration
Prerequisites: Control Flow and Functions
Before we start

What you will learn

  • Write list, dict and set comprehensions
  • Add conditional filters inside comprehensions
  • Use generator expressions for large or streamed data
Introduction

What is this?

A comprehension is a compact loop that produces a collection: [x*2 for x in data if x > 0] builds a filtered, mapped list in one expression. A generator expression uses parentheses and yields items lazily one at a time, so you can process unbounded streams and enormous files without holding them in RAM.

Motivation

Why do you need it?

  • Readability: the transformation is stated as data flow, not bookkeeping.
  • Laziness: generators make 'sum over a 50GB file' feasible without loading it.
  • One expression replaces loops that are slower to read and easier to get wrong.
  • Comprehensions compose with functions (map/filter semantics) for clean pipelines.
Think about it

Mental model

Imagine

A sorted assembly line

Think of

Comprehensions and generators

The line tells workers: 'double each item that passes the size check, ship it'. A list comprehension is the whole line built instantly; a generator is the belt that only produces the next item when the next worker reaches for it — nothing is stacked up in between.

Mechanics

How it works

  • [expr for x in seq] builds a full list in one pass, applying expr to each element.
  • A trailing 'if' filters: [expr for x in seq if cond].
  • Dict syntax swaps the target: {key: value for x in seq}.
  • Set comprehensions dedupe automatically: {x % 4 for x in range(10)}.
  • Generator expressions (genexpr) use parentheses and evaluate lazily — one item at a time on iteration.
  • sum(), any(), min(), max() consume generators without materializing the list.

Comprehensions vs generators by memory.

Sequence
Comprehension
Generator
Use
Hands-on

Examples

Same logic, eager vs lazy

Same logic, eager vs lazy
vals = range(100)
doubled = [v * 2 for v in vals if v % 5 == 0]   # eager list
total = sum(v * 2 for v in vals if v % 5 == 0)  # lazy genexpr
print(len(doubled), total)
 
names = {"austin", "houston", "dallas"}
upper = {n.upper() for n in names}
lookup = {n: len(n) for n in names}
print(upper)
print(lookup["austin"])

The list materializes for reuse; the genexpr feeds sum() and never builds storage; set and dict comprehensions transform keys cleanly.

Real world

In production

  • A log-processing job aggregates errors with a genexpr over a streamed file, holding one line at a time.
  • A data-warehouse audit builds a set of expected contract IDs and diffs it against actuals.
  • A feature-engineering step maps thousands of columns with a dict comprehension driving renaming rules.
Guidelines

Best practices

  • Keep comprehensions to one filter and one expression; anything longer is a real loop.
  • Use a generator for one-shot reductions (sum, any, max) over large inputs.
  • Name the transformation — {'US' if c == 'USA' else c for c in cols} reads better inline.
  • Consume a generator exactly once; if you need it twice, materialize the list.
Watch out

Common mistakes

  • Turning everything into a comprehension, producing unreadable one-liners.
  • Converting a huge genexpr to a list 'to be safe' and blowing up memory.
  • Reusing an exhausted generator and getting an empty result the second pass.
  • Confusing a genexpr with a plain tuple — the parentheses version is lazy.
When things break

Debugging tips

  • If a genexpr seems empty, you may already have consumed it earlier in the function.
  • A slice or index operator on a generator fails — you must materialize first.
  • When filtering drops everything, invert the condition on three sample rows to sanity-check.
  • Profile memory: a giant list comprehension shows t̶h̶e̶ peak — switch to a genexpr for reductions.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which expression is lazy?

2

A generator can be iterated more than once.

Go deeper

Official documentation

Python docs: comprehensions
docs