Streaming
Intermediate 17 min read Streaming

Stream Processing Basics

From messages crossing a pipe to state over time

Stream processing turns raw events into state: windowing groups events in time, keyed state tracks per-entity aggregates, and event-time handling decides how late data behaves. These three ideas distinguish real stream processors from message routers.

stream processingwindowingtumblingslidingsessionevent timewatermarkkeyed state
Prerequisites: Delivery Semantics
Before we start

What you will learn

  • Distinguish stateless routing from stateful processing
  • Explain tumbling, sliding and session windows
  • Use watermarks to define lateness handling
Introduction

What is this?

Stream processing computes on events while they're in motion: windowing buckets events by time (tumbling — fixed disjoint, sliding — overlapping, session — idle-gap based), keyed state holds per-key aggregates across events, and event-time semantics with watermarks decide when windows close and how late data is handled. Without these, you have message routing, not stream processing.

Motivation

Why do you need it?

  • Business questions are time-bucketed: 'orders in the last 5 minutes', 'sessions ending', 'active users this hour'.
  • Keyed state unlocks per-user/per-device reasoning — counts, sessions, flows.
  • Event-time processing keeps the numbers correct when events arrive late or out of order.
  • Windowing is how 'real-time analytics' and 'continuous alerts' actually stay meaningful.
Think about it

Mental model

Imagine

Sorting mail by delivery time

Think of

Windowing + watermark

You sort letters into the slot matching when they were sent (event time). A watermark is like a printed 'all letters up to 10:00 should have arrived': when it passes, you seal that slot (close the window) and count what's inside. Late letters after seal are processed via the late-data path (allowed lateness).

Mechanics

How it works

  • A window assigns each event to a bucket based on its timestamp and the window type.
  • Tumbling windows are fixed-length, disjoint (0-5, 5-10); sliding overlap (every 1 min, length 5); session split on idle gaps.
  • Each window aggregates its events (count, sum, top-N) emitted when windows close.
  • Watermarks advance event-time progress; events older than the watermark may be dropped or sent to a late-data path.
  • Keyed state stores per-key values between events, enabling session spreads, running totals, and per-user logic.

Tumbling, sliding and session windows.

Tumbling
Sliding
Session
Watermark
State
Real world

In production

  • Session windows on clickstreams bound each user's browsing session by 30-min idle gaps.
  • Tumbling 5-min windows track website error counts and page an on-call when p95 SSE > 1000.
  • Sliding 1-min windows give a rolling 'orders in the last hour' number for a live revenue tracker.
Guidelines

Best practices

  • Bucket on event time (when it happened) and set a watermark matched to your arrival skew.
  • Bound keyed state: cap keys and TTLs so unbounded distinct keys don't sink memory.
  • Choose the smallest window your SLA needs — smaller windows = more updates, more cost.
  • Handle late data explicitly: allowed-lateness, or a correction/compaction path.
Watch out

Common mistakes

  • Windowing on processing time — numbers drift by ingestion delay and jitter.
  • Infinite session windows that never close (no idle-timeout).
  • State keyed by a high-cardinality unbounded value (event_id) → OOM.
  • Dropping late events silently, then under-counting a critical metric.
Root cause analysis

Common issues

Error
Metrics are consistently 5 minutes behind the wall clock
Root cause

Windowing on ingestion time, or watermark too pessimistic

Fix

Window on event time with a watermark of real lateness; tune separate monitoring.

Error
Memory explodes on a session-window job
Root cause

Unbounded keys or sessions never closing due to a huge idle timeout

Fix

Cap session gap, key cardinality and state TTL thresholds.

Error
Late events create numbers that drift backward on re-run
Root cause

Allowed-lateness reprocessing without a correction/compaction step

Fix

Emit corrected totals, or exclude late data and document the cutoff.

When things break

Debugging tips

  • Check event time vs processing time to diagnose 'numbers always lag' complaints.
  • Inspect the watermark vs the newest event time — wide gaps mean late-path hazards.
  • State growth → look at keys cardinality and TTL settings.
  • Double-counting across windows → inspect window overlap (sliding data counted twice by design).
Interviews

Interview checkpoint

Practice

Hands-on lab

Implement Tumbling, Sliding and Session Windows
35 min · intermediate
In progress
Objectives
  • Classify a stream of events into tumbling, sliding and session windows on event time
  • Observe how window type changes bucket membership and emitted counts
  • Apply a watermark to decide when late events stop entering a closed window

Implement Tumbling, Sliding and Session Windows

35 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A 5-minute tumbling window with a 2-minute watermark finalizes an event-time 12:03 window at roughly...

2

Sliding windows can count the same event in more than one window.

Go deeper

Official documentation

Flink — event time & windows
docs
Spark — windowed aggregations
docs