Streaming Fundamentals

• Describe producer → broker → consumer architecture
• Name the four properties streams trade off: freshness, ordering, replay, delivery
• Explain what an offset is and why replay matters

• Streaming meets freshness: sub-second reaction to events (fraud, alerts, live dashboards).
• Brokers decouple producers from consumers — neither blocks on the other's speed.
• Replay lets you re-read historical events to rebuild state or re-run logic.
• Modern stacks push freshness where batch simply can't follow (updates to products, telemetry).

Send a letter (event) to the mailroom (topic). Then every department subscribed (consumer) can read it; the mailroom keeps a copy for re-delivery (replay), and each department tracks which letters it has read (offset).

• Producers publish events to a topic — a partitioned log with an append-based sequence.
• The broker stores the events durably and serves them to consumers on demand.
• Consumers read a stream of events (pull-based in Kafka) and track an offset per partition.
• Groups of consumers share a topic's partitions so throughput scales horizontally.
• Ordering holds within a partition; delivery can be at-least-once, exactly-once or at-most-once.

• Clickstream events are published to Kafka and consumed by analytics, dashboards and ML features all from one topic.
• Order messages go through a stream so inventory, shipping and finance each react asynchronously to the same event.
• A monitoring pipeline streams server metrics and triggers alerts within a second of an anomaly.

• Keep the broker a fast buffer: move heavy transforms out of the streaming path or into consumers.
• Always replay first when debugging — reprocessing a topic is the universal truth-check.
• Set retention (time or size) so replay windows match you recovery SLAs, not laziness.
• Design events as small, schema-versioned records — not blobs.

• Publishing enormous payloads, then wondering why the broker is the bottleneck.
• Treating the topic as a queue to delete after consumption, losing the ability to replay.
• Ignoring ordering guarantees and building 'ordered calculations' on cross-partition data.
• Using exactly-once as a blanket requirement when at-least-once + idempotent sinks suffices.

• Check consumer lag (end offset minus consumer offset) first — it tells you who's falling behind.
• Compare producer timestamps and consumer timestamps to isolate end-to-end latency.
• Replay a window of the topic and recalc totals — if results match, upstream logic is fine.
• Inspect a single event's schema and content before blaming the pipeline for wrong metrics.