Python for Data Engineering
Intermediate 15 min read Python for Data Engineering

Generators and Iterators

Process unbounded data with bounded memory

Generators yield values lazily, so you can process data that does not fit in memory — the foundation of streaming code and chunked reads in every data tool.

generatoriteratorlazy evaluationyieldmemorystreaming
Prerequisites: Python Data Structures
Before we start

What you will learn

  • Write generators and generator expressions
  • Explain laziness: nothing computes until iterated
  • Chain generators instead of buffering intermediate lists
Introduction

What is this?

A generator is a function with `yield` that produces values one at a time on iteration, instead of building a full list. Combined with iterators, they make 'process one record at a time' possible — the reason Python can read a 100 GB file or an infinite event stream.

Motivation

Why do you need it?

  • A list comprehension over a 10M-row file materializes 10M objects in memory; a generator expression streams them.
  • Transforms in pandas/Spark and every streaming connector are implemented with lazy producers/consumers.
  • Chaining lazy operations keeps memory flat regardless of input size.
Think about it

Mental model

Imagine

A tap vs a bucket

Think of

Generator vs list

Building a list fills the whole bucket before you drink; a generator turns the tap on and you sip as it flows. Both deliver water, only one scales to an infinitely large tank.

Mechanics

How it works

  • `def gen(): yield 1; yield 2` suspends execution after each yield until the next `next()` call.
  • Generator expressions `(x*2 for x in big)` are the lazy sibling of list comprehensions.
  • Iteration drives a generator: each `for`/`next()` resumes after the last yield.
  • Generators can be chained — a pipeline of small generators mirrors a DAG of transformations.
  • `itertools` (islice, chain, groupby) composes with generators for production flows.

A lazy transformation chain.

Read lines
Parse
Filter
Aggregate
Done
Hands-on

Examples

Streaming a large file's first N valid rows

Streaming a large file's first N valid rows
import itertools
def clean_records(path):
    with open(path) as f:
        for line in f:
            parts = line.strip().split(",")
            if len(parts) == 3:  # valid
                yield parts
first_10 = list(itertools.islice(clean_records("big.csv"), 10))

The file is read lazily; islice stops pulling after 10 valid rows — gigabytes can be skipped without loading them.

Real world

In production

  • A click-parser yields parsed events from a log stream rather than buffering a day's events.
  • A chunked CSV reader splits a huge file into pandas DataFrames of N rows at a time for bounded-memory writes.
  • An extractor wraps an API's pagination in a generator so callers iterate pages without knowing pagination exists.
Guidelines

Best practices

  • Return early: a generator that must produce a list should collect and return, or callers re-iterate is impossible.
  • Close underlying resources properly — generators opened inside `with` keep the file handle until exhausted.
  • Use `itertools.islice(gen, n)` to bound consumption and 'peek' a few rows.
  • Prefer generator expressions for single-pass filtering/mapping of large sequences.
Watch out

Common mistakes

  • Assuming generators can be re-iterated — they are single-use; `.list()` them if you need two passes.
  • Opening a file inside a generator and never consuming it fully, leaking the handle.
  • Wrapping a generator in a function that returns it but forgetting to call it — returning a generator object, not values.
Root cause analysis

Common issues

Error
Generator produces nothing when I iterate twice
Root cause

Generators are exhausted and cannot be restarted

Fix

Re-create the generator (call the function again) or materialize with list() once if you need two passes.

Error
Memory grows despite using a generator
Root cause

An intermediate list(...) or comprehension materializes the sequence

Fix

Convert to generator expressions or itertools so the chain stays lazy.

When things break

Debugging tips

  • Confirm laziness: put a print in the generator body; nothing prints until you iterate.
  • If a 'streaming' step spikes memory, check that an intermediate `list(...)` is not materializing the whole input.
  • Trace a pipeline by iterating with a bounded islice and printing one record.
Interviews

Interview checkpoint

Practice

Hands-on lab

Stream a 10M-Row File Without Memory Blow-Up
25 min · intermediate
In progress
Objectives
  • Generate a large CSV without loading it into memory
  • Parse and filter it with a generator chain
  • Prove memory stays flat and the output is correct

Stream a 10M-Row File Without Memory Blow-Up

25 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

What happens when you run `list(x * 2 for x in range(1000000))`?

2

A generator can be iterated multiple times like a list.

Go deeper

Official documentation

Iterators and generators — Python HOWTO
docs
itertools — Python docs
docs