Batch Inference
Score the whole table, calmly
Batch scoring is the workhorse for bulk decisions: pull data, score in parallel, write results. Orchestrated as scheduled jobs with idempotency, watermarking and failure retries — cheap, auditable, deterministic.
What you will learn
- Design batch jobs: input slices, idempotency, ordering
- Make failures retryable and results auditable
- Keep batch/online scoring on the same features
What is this?
Batch inference scores a dataset on a schedule (nightly, hourly) instead of per-request. Jobs pull the input partition (e.g. customers updated since watermark), run the model over rows in parallel, and write predictions back with metadata (job id, model version, run timestamp). Idempotency means re-runs overwrite the same keys without side effects; failure handling retries failed partitions; and every output row carries lineage for audits.
Why do you need it?
- Massively cheaper per prediction than online at equal volume.
- Deterministic and testable — a batch run is reproducible on the same data + version.
- Fits upstream data cadence: daily features → daily scores.
- Simple to justify: one pipeline, one manifest, one retry policy.
Mental model
A mailroom sorting the night's letters by a route sheet
Scheduled batch scoring
Every night the same clerk (the job) opens the same mailbag (input partition), stamps each letter with the route — and if a bag tears, re-ran only that bag, never the whole post office. Letters sorted yesterday don't get re-done; the route sheet (job manifest) records which bags were done when.
How it works
- Define the input: table + partition predicate (updated_at ≥ watermark).
- Score in parallel; rows are independent → embarrassingly parallel across workers/partitions.
- Write outputs with a key: (business key, job id, model version) — idempotent by key.
- Orchestrate: airflow/dagster/scheduler triggers on data readiness + monitors freshness.
- Retry semantics: failed partitions retried; successes never re-scored; results row-pinned to version.
Scheduled job: pull, score, write with lineage.
In production
- Nightly credit-risk portfolio scores into the data warehouse.
- Hourly churn-risk scores feeding campaign systems.
- On-demand re-scores of a whole segment after policy/data changes.
Best practices
- Make runs idempotent: same input → same output rows, safe to re-run.
- Watermark + partition: only score changed rows; keep a 'last scored' registry.
- Log a job manifest (input range, model version, row count, duration) for every run.
- Reuse feature code with the online path to avoid skew.
Common mistakes
- Re-scoring everything each run (cost grows with the whole table, not the delta).
- Non-idempotent writes — a retry appends duplicates.
- Unversioned outputs breaking downstream joins.
- No row-count/freshness check, so a silently empty run ships 'no scores'.
Common issues
Retry produced duplicatesWrite wasn't idempotent by key
Upsert on (business key, model version); never append blindly.
Empty output despite big inputFilter predicate bug or watermark drift
Assert row counts (input vs output delta); alert on mismatch thresholds.
Job costs explode weeklyFull-table rescore with no watermarking
Partition by update watermark; score only the delta; archive full rescans to on-demand only.
Debugging tips
- Diff two consecutive runs on overlapping rows; deltas should match ONLY where inputs changed.
- Alert on row-count sudden drops to zero — a silent-empty batch is a production disaster.
- Time the job at max historical partition; failure-driven boundary > happy-path boundary.
- Correlate batch output with online scores on a shared sample as a permanent skew check.
Interview checkpoint
Quick quiz
Idempotent batch writes mean:
Batch jobs should score the entire table every run for freshness.