Batch Inference
Scoring the world in windows, efficiently
Batch inference scores lots of rows on a schedule rather than on demand: load the model, process a window of data, write results to storage. It's the cost-efficient workhorse when nobody is waiting for a single prediction.
What you will learn
- Describe when batch inference beats online serving
- Structure a batch scoring job: load, process, write
- Track batch runs as versioned, monitored operations
What is this?
Batch inference processes large sets of inputs in scheduled jobs: a job loads the model and a data window, generates predictions for every row, and writes them (or triggers downstream actions) to storage. It trades real-time latency for throughput and cost efficiency.
Why do you need it?
- Not every prediction has someone waiting — scoring 10M customers' churn doesn't need sub-second answers.
- Batch compute is dramatically cheaper per prediction than resident servers.
- Scheduled runs are auditable: same input version → same scored window.
- Fits naturally into data warehouse workflows and feature-store pipelines.
Mental model
A chemist running samples in a lab overnight
Batch inference
Each sample (row) takes time, but the lab processes racks at once (batch), reports by morning (schedule), and the results sit in a binder/table (output store) for whoever reads them. Nobody stands waiting on one test tube.
How it works
- A trigger fires: schedule, new data version, or downstream request.
- The job reads a pinned dataset window with its schema and version.
- The model is loaded once and applied across rows, often in chunks.
- Predictions plus metadata (model version, window, run id) write to output storage.
- The run is logged: row counts, timing, quality metrics, output location.
A batch inference run.
In production
- Nightly churn scoring writes per-customer probabilities for the campaign tool.
- Loan eligibility is re-scored weekly as new EOD balances land in the warehouse.
- A recommender re-ranks the full catalog offline each hour and serves from a precomputed index.
Best practices
- Pin the input data version and model version for every run.
- Record run metadata (counts, timing, output path) before/after scoring.
- Chunk large sets to bound memory and enable retry of partial failures.
- Route okay failures to a dead-letter path instead of dying mid-run.
Common mistakes
- Re-reading the model per row — loading once is the whole point.
- Overwriting last run's output without versioning the window.
- Treating batch as fire-and-forget — no run log, no alert on empty output.
- Silently skipping bad rows, then wondering why counts don't match.
Debugging tips
- Output row count ≠ input count — check skip/error handling and dead-letter records.
- 'Model scored yesterday's data' — the trigger may have raced the data pipeline's commit.
- Chunk memory blows — reduce chunk size; GPU-vs-CPU placement may also matter.
Interview checkpoint
Quick quiz
Batch inference is preferable when…
A batch job should record input and output row counts for every run.