Big Data
Intermediate 15 min read Big Data

Spark Structured Streaming

Streams that read like DataFrames

Structured Streaming treats streams as incremented DataFrames: the same Spark SQL you use on files works on micro-batches. Windowing, watermarks and output modes control exactly how late and partial results behave.

spark streamingstructured streamingmicro-batchwindowingwatermarkkafka
Prerequisites: Spark SQL and DataFrames
Before we start

What you will learn

  • Explain the micro-batch model vs true event-at-a-time engines
  • Use event-time windows and watermarks to set result completeness
  • Pick an output mode for a dashboard vs a sink
Introduction

What is this?

Spark Structured Streaming processes unbounded tables: every micro-batch is a new snapshot of data to which you re-run your query. Since it derives each incremental result through the DataFrame API, the same SQL, aggregations and joins work. Event-time windows plus watermarks control how late data updates results, and output modes decide what to emit.

Motivation

Why do you need it?

  • Reusing the DataFrame/SQL API means one mental model for batch and stream.
  • The micro-batch model inherits exactly-once processing from Spark reliability.
  • Output modes handle the 'does the dashboard update or append?' question declaratively.
  • For stream-join and refresh-heavy cases, it removes per-event boilerplate.
Think about it

Mental model

Imagine

Reading a newspaper that updates every few seconds

Think of

Structured streaming (micro-batch)

You don't see each word as it's typeset; you see snapshots as the edition rolls off the press. You tolerate a few seconds of delay for dramatically simpler reliability.

Mechanics

How it works

  • A streaming source (Kafka, files, socket) is registered as an input DataFrame.
  • Spark processes it in micro-batches: collect new offsets, run the query, write the result.
  • Window aggregations group by event time; watermarks drop/auto-detect data older than the per-window advanced watermark.
  • Output modes decide semantics: append (only new rows), update (changed rows), complete (full result table).
  • The checkpoint directory stores offsets and state for exactly-once recovery.

Micro-batch stream processing.

Kafka/files
micro-batches
Windowed SQL
State
Sink
Real world

In production

  • A clickstream job aggregates session counts into 5-minute tumbling windows written to a Kafka topic for dashboards.
  • Orders from Kafka joined with a dimension table produce enrichened events written to Parquet with exactly-once recovery.
  • A monitoring pipeline outputs 'append' to a console/metric sink so late records don't rewrite history.
Guidelines

Best practices

  • Use event-time, not processing-time, in windows; set a watermark that matches your freshness tolerance.
  • Enable checkpointing to a stable location and version your schema carefully.
  • Choose append/update over complete for unbounded aggregate tables (complete costs memory).
  • Underprovision rate and partition align: Kafka partitions ↔ Spark task parallelism.
Watch out

Common mistakes

  • Building windows on processing time — numbers drift and late events corrupt history.
  • A watermark so tight that legitimate late events are dropped silently.
  • Using complete output mode for an ever-growing windowed aggregation (memory blows).
  • Not checkpointing, so a crash replays duplicates or loses state.
Root cause analysis

Common issues

Error
Dashboard numbers never change after the first batch
Root cause

Watermark/event time handling puts every record out of window

Fix

Verify the column Spark uses for event time is actually the source event timestamp (not ingestion timestamp).

Error
Restart replays old offsets
Root cause

Checkpoint missing or on a different location

Fix

Point the checkpoint directory to the same stable path (S3/DBFS) across restarts.

Error
Memory grows without bound
Root cause

Complete-mode update of an unbounded key set

Fix

Use update/append mode or add a watermark + retention on the state store.

When things break

Debugging tips

  • Check the batch progress and the streaming query's lag: input row counts vs output rows.
  • If results don't update the dashboard, inspect watermark vs event time ranges.
  • Drop counts? Revisit the window duration and watermark before blaming the sink.
  • Use the `lastProgress`/`recentProgress` stream query details to see per-batch latency.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

An aggregate table that rewrites every changed row to the sink uses which output mode?

2

Watermarks are only useful if your streams have late data.

Go deeper

Official documentation

Structured Streaming programming guide
docs
Structured Streaming + Kafka guide
docs