Data Engineering Foundations
Beginner 14 min read Data Engineering Foundations

Batch vs Streaming

Freshness is a spectrum, not a binary

Batch processing moves data on a schedule while streaming moves it continuously with low latency. Each has different SLAs, failure semantics and cost profiles.

batch processingstreaminglatencyevent timeKafkaSpark
Prerequisites: The Data Lifecycle
Before we start

What you will learn

  • Decide batch vs streaming from business latency requirements
  • Explain the failure-model differences between the two
  • Recognize hybrid patterns and when they make sense
Introduction

What is this?

Batch processing collects data over an interval (daily, hourly) and processes it all at once, often at a fixed time. Streaming processes events continuously as they arrive, usually within seconds. The choice is governed by two questions: how fresh must the data be, and how much complexity can you afford?

Motivation

Why do you need it?

  • Freshness drives decisions — a fraud alert at +24h is worthless, a sales report at +24h is fine.
  • Batch is dramatically simpler: run a job, it's either done or failed, retry cleanly.
  • Streaming gives you near-real-time dashboards, alerts and feature serving, but adds exactly-once headaches, backpressure and state.
  • Most warehouses still land batch; streaming feeds it in micro-batches.
Think about it

Mental model

Imagine

Mail vs text messages

Think of

Batch vs streaming

Mail is collected, sorted and delivered on a schedule — cheap, simple, fine for non-urgent deliveries. Text messages arrive the moment they are sent — complex infrastructure under the hood, but needed when timeliness is the point.

Mechanics

How it works

  • A batch job reads a bounded chunk (file, table partition or day's events) and processes it start to finish.
  • Orchestration triggers the job on schedule; on failure it can retry or re-run the whole window.
  • A streaming pipeline consumes an unbounded log of events from a broker like Kafka.
  • Stream processing aggregates with windowing (tumbling, sliding, session) and keeps state for joins or counts.
  • Delivery semantics (at-least-once, exactly-once) gate how much duplicate handling downstream must do.

Batch vs streaming data paths.

Sources
Batch: scheduled pull
Stream: continuous pull
Warehouse / lake
Consumers
Real world

In production

  • Nightly batch reconciles banking ledgers — correctness matters more than seconds.
  • Streaming detects fraudulent card swipes in under 2 seconds using a windowed count model.
  • A hybrid: streaming loads fresh events into the lake hourly micro-batches, then a nightly batch builds the final modeled tables.
Guidelines

Best practices

  • Work from the latency SLA backward: if +15 minutes is fine, don't build streaming.
  • In batch, always process idempotently so re-runs are safe.
  • In streaming, choose delivery semantics explicitly and document duplication guarantees.
  • Use event time (when it happened) rather than processing time (when you saw it) for windowing.
Watch out

Common mistakes

  • Building a streaming pipeline for a daily report — complexity with zero payback.
  • Assuming stream events arrive ordered or exactly once — they don't by default.
  • Forgetting late data arrives after windows close, causing silent underestimates.
  • Wiring batch tools to a stream and silently dropping events on bursty days.
Root cause analysis

Common issues

Error
Streaming dashboard numbers drift backward
Root cause

Late-arriving events were reprocessed into closed windows

Fix

Configure watermark/late-data handling or accept 'final after N minutes of lag' semantics.

Error
Nightly batch misses a day occasionally
Root cause

Job runs at fixed time but source wasn't ready, and there is no retry

Fix

Add dependency sensors plus retries, and re-run windows when the source catches up.

Error
Duplicates appear in the warehouse from the stream
Root cause

At-least-once delivery replayed events after a crash

Fix

Dedupe downstream with a primary key + window (publish_minus_processing) or use exactly-once sinks.

When things break

Debugging tips

  • For batch lag, check orchestration status and the last successful run timestamp first.
  • For stream lag, inspect consumer group offsets vs the broker's end offset.
  • If late events keep changing numbers, check whether the windowing uses event time and handles late/Kafka-sidewater marks.
  • Compare counts 'in (source)' vs 'out (target)' to prove whether events are being dropped or duplicated.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

A fraud alert must fire within 3 seconds of a transaction. Which approach fits?

2

Streaming always produces data exactly once, in order.

Go deeper

Official documentation

Kafka docs — exactly once
docs
Flink — event time & watermarks
docs