Request Batching

• Understand where batch APIs discount cost
• Identify latency-flexible workloads
• Aggregate independent calls into batches

• A summary job over 10,000 documents at interactive prices is needlessly expensive.
• Providers discount batch to move load to low-demand windows.
• Fewer, larger requests amortize overheads (auth, queueing, network) too.

You could courier every parcel same-day (each costly), or wait a few hours and send a container at a volume rate. Same items, way lower bill.

• Write deferred tasks (summary, extraction, backfill) to a job queue.
• Submit them to the provider's batch endpoint (or your own batcher with FIFO windows).
• Poll for results; retry failed items; attribute cost per job.
• Reserve interactive pricing only for requests where latency is visible.

• A nightly content-enrichment job runs 50,000 summaries via the batch API.
• A migration backfills embeddings in batches with a few-day SLA.
• An internal Q&A index refreshes its digest jobs overnight at the discount rate.

• Separate interactive and batch traffic at the gateway so the right pricing applies.
• Give each submission a batch id and each item its own id for result matching.
• Set up retry and dead-letter handling per item, not per batch.
• Track batch cost vs saved dollars explicitly to justify the latency trade.

• Batching latency-sensitive requests to save cents — users feel every minute.
• No per-item result matching, so a partial failure poisons the whole run.
• Forgetting that batch SLAs expire; plan result collection well inside the window.

• Batch results missing → poll by batch id and item id; treat expiry as a failure case.
• Discount not materializing → confirm you're on the batch endpoint, not the sync one with quotas.
• Unusable results → check the model's max_tokens and per-item params were set per job.