Batch Pipeline Design
Bounded steps that run nightly without drama
A batch pipeline is a sequence of bounded, idempotent steps with explicit windows: read → validate → score → write → log. Design decisions are chunking, retry, and output immutability — the details that keep a nightly job honest for years.
What you will learn
- Decompose a batch job into bounded, idempotent steps
- Handle partial failures with retry and dead-lettering
- Make re-runs safe via windowed, immutable outputs
What is this?
Batch pipeline design decomposes scoring into small, restartable steps — read window, validate, score in chunks, write outputs, log run — where each step is idempotent and window-scoped. Re-runs are safe; partial failures are retried, not feared.
Why do you need it?
- One giant script either works fully or fails obscurely; bounded steps fail loudly and recover.
- Idempotent steps mean 'run it again' is an answer, not a risk.
- Window-scoped data keeps a re-run from reprocessing the world.
- Auditable runs are the difference between 'numbers look off' and knowing which window did it.
Mental model
A long-distance run broken into water stations
Batch pipeline steps
You don't finish by sprinting once; you pass stations (steps) that log progress. If you collapse at kilometer 25 you resume at 25, not back at the start line — stations make the journey restartable and observable.
How it works
- Read: pull the pinned window (e.g. yesterday's rows) by its data version.
- Validate: schema, counts, freshness against the manifest.
- Score: chunk the input, apply the loaded model, keep per-chunk progress.
- Write: emit predictions to a windowed output path (immutable, versioned).
- Log: record run id, counts, duration, output location.
Bounded, restartable batch steps.
In production
- A nightly job scores 'yesterday' by window = date; a bad Monday re-runs only Monday.
- A dead-letter table holds rows that fail schema, counted per run.
- Chunked scoring lets a Spark job checkpoint so a crash resumes from the last saved chunk.
Best practices
- Make every step write into a windowed, immutable location keyed by run id.
- Validate counts and schema before scoring, not after.
- Chunk large inputs and commit progress so retries skip done work.
- Log the run boundary before AND after scoring.
Common mistakes
- A single 'do everything' script that dies mid-way and can't resume.
- Outputs that overwrite last night's, destroying the comparison window.
- Re-reading the model per entity instead of loading once.
- No dead-lettering: one bad row fails the whole window silently by skip.
Debugging tips
- Counts mismatch — compare run-input vs output row count; the gap is in skip/error handling.
- Re-run produced a different result — confirm the window and seed didn't drift; idempotency holds only if steps are.
- Job 'hung' — the scoring chunk may be unbalanced; look for a data skew hotspot.
Interview checkpoint
Hands-on lab
- Decompose scoring into restartable steps
- Add chunk progress and dead-letter handling
- Prove a re-run is idempotent
Design a Restartable Batch Job
35 min · intermediate
Quick quiz
The key property making a batch step safe to retry is…
Batch jobs should record counts before and after to catch silent skips.