Streaming
Advanced 17 min read Streaming

Streaming Architectures and Patterns

Composing the patterns into a real platform

Real systems mix patterns: event sourcing, CQRS, change data capture, fan-out/fan-in, lambda and kappa architectures. The design is about choosing where state lives and how consistency degrades.

event sourcingCQRSlambda architecturekappa architectureCDCfan-outstate
Prerequisites: Stream Processing Basics
Before we start

What you will learn

  • Choose between lambda and kappa architectures
  • Apply event sourcing + CQRS where it fits
  • Recognize fan-in/fan-out and CDC patterns
Introduction

What is this?

Streaming architect patterns repeat across production systems: event sourcing stores truth as an event log; CQRS separates read models from write path; CDC streams DB changes; fan-out/fan-in distribute and collect; windowed microservices keep per-entity state. Kappa architecture does everything through a stream (log as truth), while lambda layer a batch + speed path over the same sources.

Motivation

Why do you need it?

  • Choosing the pattern is choosing your consistency + complexity trade early.
  • Kappa's stream-as-truth simplifies ops where replay suffices.
  • Event sourcing gives the auditability and replay that regulatory systems demand.
  • Knowing the patterns means recognizing 'this is just a fan-in over a session window' instead of inventing something custom.
Think about it

Mental model

Imagine

Choosing the blueprint before pouring concrete

Think of

Picking an architecture pattern

A hospital, a warehouse and a theater have very different structures even though plumbing is similar. You pick the layout from the requirements — then the pipes (Kafka, Flink, state) go where they must.

Mechanics

How it works

  • Event sourcing: every state change is appended to an event log; current state = replay of events (or snapshots + tail).
  • CQRS: writes go to the event/命令 model; reads come from projectors maintaining read-model tables.
  • CDC: DB WAL/binlog → events, feeding search, warehouse, caches without writes to the source.
  • Fan-out: one event topic consumed by many independent systems (each its own consumer group).
  • Kappa: single streaming path processes everything; lambda runs a batch (fast correctness) + speed (freshness) layer.

Kappa vs lambda architecture.

Sources
Kappa: stream all
Lambda: batch+speed
State
Consumers
Real world

In production

  • A fintech models ledger as event sourcing: replay reconstructs any account's exact state for audit.
  • A marketplace syncs product search read-models via CDC; CQRS separates the write system-of-record from reads.
  • A media platform uses kappa-style streaming for real-time metrics, keeping the source-of-truth log replayable.
Guidelines

Best practices

  • Prefer kappa unless you have a hard batch-quality requirement tiers can't cover.
  • Use event sourcing where replayability/audit is a requirement, not everywhere.
  • Keep the event log the single source of truth; derived read models are disposable.
  • Design fan-out for independence: each consumer owns schema evolution and can lag independently.
Watch out

Common mistakes

  • Abusing event sourcing where the current state is small and simple (cost > value).
  • Letting two pipelines in lambda drift apart — same source, different semantics.
  • Storing commands in the event log; writing derived state into the truth log.
  • Ignoring schema evolution when many read models consume one event.
Root cause analysis

Common issues

Error
Read model diverges from the event log periodically
Root cause

A projector missed or duplicated an event (commit bug)

Fix

Replay the projector from offset 0 and automate reconciliation audits.

Error
Two lambdas produce slightly different numbers
Root cause

Speed and batch versions use different aggregate windows/event definitions

Fix

Unify event definitions; use a single replayable definition of 'final'.

Error
Global state store too slow for lookups
Root cause

Read path entirely on a single queyable state vs CQRS read-side

Fix

Project a read-model table (via CDC/events) and query that store directly.

When things break

Debugging tips

  • Reconcile read models by re-replaying: if the projector reproduces the table, state is derived correctly.
  • Kappa/lambda drift → compare last-committed-offset vs last-served event per read model.
  • CDC gaps → verify the WAL retention vs projector lag.
  • Fan-out divergence → per-consumer schemas vs registry; most such bugs are schema, not logic.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The pattern that stores truth as an append-only event log is called...

2

In kappa architecture, the log is treated as the single source of truth.

Go deeper

Official documentation

Kappa architecture (Jay Kreps)
docs
CDC patterns guide (Debezium)
docs