Mini-Batch and Windowed Processing

• Place mini-batch between pure batch and streaming
• Choose window size, watermark and trigger semantics
• Bound correctness vs freshness tradeoffs near real-time

• Not everything needs sub-second answers, but nightly is too stale for churn or fraud intent.
• Mini-batch reuses batch tooling (Spark, Flink) with bounded windows.
• It's easier to retry and audit than true streaming.
• Throughput stays high because you still amortize model loads over windows.

Kids arriving in the window (cars) board one bus (batch). If you set the schedule short, kids wait less but buses run almost empty; long, and they wait long. The route (window + trigger) sets the freshness/cost tradeoff.

• Data is grouped by a window boundary (e.g. every 5 minutes by event time).
• A watermark marks how late data may be before the window closes.
• When the trigger fires, the window's rows are scored and written.
• Late rows after watermark go to a side channel for reconciliation.
• Scores are written per window, addressable and monitored like batch runs.

• A fraud system scores 5-minute windows of transactions, catching patterns that nightly misses.
• A forecasting job emits hourly windows that a streaming dashboard reads.
• A recommendation service re-scores trending items every 10 minutes from a mini-batch join.

• Choose window size from the business freshness, then cost it.
• Set watermarks deliberately — too tight drops late data, too loose delays output.
• Make triggers explicit: processing-time vs event-time windows.
• Record window boundaries in output metadata for reconciliation.

• Window size chosen by cluster capacity, not freshness need.
• Ignoring watermarks — late events silently vanish, and scores look 'off' later.
• Treating mini-batch as streaming and debugging with streaming assumptions.
• Writing overlapping windows, double-counting entities.

• Scores lag wall-clock — event-time vs processing-time skew; align your watermark/trigger.
• 'Missing' rows — they arrived past the watermark; check the late channel, not the window.
• Outputs differ from nightly — windowed aggregation at 5-min vs nightly sees different partial states.