System Design
Intermediate 16 min read System Design

Message Queues & Event Streaming

Message queues decouple producers from consumers: a producer publishes a message and moves on, while consumers process it at their own pace. Queues guarantee at-least-once delivery, which forces idempotent consumers, and streams like Kafka add durable, replayable, ordered logs. Lags, ordering and poison messages are the operational realities of every queue-backed system.

message queuepub/subkafkaconsumer groupoffsetat-least-onceidempotentDLQdead letterevent streaming
Prerequisites: Scalability Basics Event-Driven Architecture Database Scaling: Replication & Sharding
Introduction

What is this?

A message queue is a broker that accepts messages from producers, stores them, and hands them to consumers, so the two never meet synchronously. Point-to-point queues deliver each message to one consumer; pub/sub topics broadcast to many; streaming platforms like Kafka add durable, ordered, replayable logs partitioned for parallelism. Brokers deliver with at-least-once semantics by default, so consumers must be idempotent, and failures route to a dead-letter queue (DLQ) instead of blocking the pipeline.

Motivation

Why do you need it?

  • Asynchronous decoupling lets a slow or briefly-down consumer never block the producer's request path.
  • Queues absorb bursts: when traffic spikes, messages buffer and drain at the consumer's speed instead of dropping work.
  • Consumers scale independently — add workers to process faster without touching the producer.
  • Retries and DLQs make failures visible and recoverable instead of silently lost.
  • Event streams preserve a durable, replayable history that can rebuild state, feed analytics, or power new consumers later.
Think about it

Mental model

Imagine

A restaurant kitchen and waiters shouting orders at each other

Think of

A ticket rail with numbered orders

The cook drops a ticket on the rail and moves on; any free waiter takes it. If a ticket is illegible it goes to a special bin (the DLQ) rather than blocking the whole kitchen. Customers can also subscribe to the daily specials board (pub/sub) and react independently.

Mechanics

How it works

  • A producer sends a message to a queue or topic on the broker; the send returns once the broker has accepted it.
  • The broker persists the message durably and delivers it to consumers, either pushing or on poll.
  • Queues hand each message to one consumer in the group; topics fan out a copy to every subscribed consumer group.
  • Consumers acknowledge after processing; unacknowledged messages are redelivered, giving at-least-once semantics.
  • Streams additionally divide data into partitions; messages with the same key go to the same partition, preserving per-key order.
  • Messages that exhaust their retry budget land in a dead-letter queue for inspection and manual replay.

A producer publishes once; the broker persists and fans out; one consumer per partition in a group processes it.

Order service
Publish
Broker
Consumer group
Process + commit
DLQ
Reference

Key commands

kafka-topics.sh --create --topic order.created --partitions 3 --replication-factor 1

Create a topic with three partitions for parallel consumption.

Created topic order.created.
kafka-console-producer.sh --topic order.created --property key.separator=:

Send test messages interactively with explicit keys.

kafka-console-consumer.sh --topic order.created --group payments --from-beginning

Consume from the beginning of the topic in the payments group.

kafka-consumer-groups.sh --describe --group payments

Show lag per partition — how far consumers are behind the latest messages.

TOPIC          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG\norder.created  0          1234            1240            6
kafka-topics.sh --describe --topic order.created

Inspect partition count, replication and leaders for a topic.

docker compose exec kafka kafka-consumer-groups.sh --group payments --reset-offsets --to-earliest --execute

Reset consumer offsets to replay messages (use carefully in production).

Hands-on

Examples

A Kafka producer and consumer for order events (KafkaJS)

A Kafka producer and consumer for order events (KafkaJS)
import { Kafka } from "kafkajs";
 
const kafka = new Kafka({
  clientId: "order-service",
  brokers: ["kafka-1:9092", "kafka-2:9092"],
});
 
// Producer: publish the fact that an order was created
const producer = kafka.producer();
await producer.connect();
await producer.send({
  topic: "order.created",
  messages: [
    { key: "order-123", value: JSON.stringify({ orderId: 123, total: 49.99 }) },
  ],
});
 
// Consumer: a payments worker in a consumer group
const consumer = kafka.consumer({ groupId: "payments" });
await consumer.connect();
await consumer.subscribe({ topic: "order.created" });
await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const event = JSON.parse(message.value.toString());
    // Idempotent: charging the same orderId twice is a no-op
    await chargePayment(event.orderId, event.total);
    console.log("paid", event.orderId, "from partition", partition);
  },
});

The producer keys the message by orderId, so every message for order-123 lands in the same partition and is processed in order. The consumer joins the 'payments' group, so if two workers run, each owns different partitions. chargePayment is written to be idempotent — charging the same orderId twice is harmless — because Kafka redelivers messages when a consumer crashes after processing but before committing its offset.

Real world

In production

  • Order pipelines publish order.created and separate workers handle payment, inventory, emails and analytics, each scaling on its own.
  • Change data capture (Debezium) streams database row changes into Kafka so search indexes and read models stay in sync with the source of truth.
  • Analytics platforms buffer millions of events through a stream, then batch them into warehouses at controlled rates.
  • Integrations use a retry queue plus a DLQ: messages that fail five times with external providers park in a DLQ where ops replays them after fixing the root cause.
Guidelines

Best practices

  • Make consumers idempotent: process by a natural key (orderId, paymentId) so at-least-once redelivery is harmless.
  • Use a consumer group for parallel processing and key messages by the entity whose order must be preserved.
  • Monitor consumer lag and alert when it trends upward — lag is the health metric of the whole pipeline.
  • Validate and schema-register messages so a bad producer cannot poison every consumer.
  • Never write to the database and publish inside one transaction without an outbox; publish only after the commit, or use the transactional outbox pattern.
Watch out

Common mistakes

  • Assuming exactly-once delivery, then getting duplicate charges or duplicate side effects when a consumer crashes.
  • Keying by timestamp or random value, which destroys the per-entity ordering you actually need.
  • Ignoring consumer lag until a backlog of hours builds up and late messages become worthless.
  • Reading from the beginning on every restart, re-processing the whole history instead of resuming from the committed offset.
Root cause analysis

Common issues

Error
Duplicate email/payment despite a single user action
Root cause

At-least-once delivery redelivered a message after a consumer crash, and the consumer was not idempotent.

Fix

Key the work by a business id and deduplicate in the consumer (check an existing record before acting), or store processed ids in a dedupe store.

Error
Consumer lag keeps growing and messages are hours late
Root cause

Too few consumers or partitions for the volume, or each message does slow work (API calls, big joins).

Fix

Add partitions and consumers, parallelize the slow step, and consider batching multiple messages per unit of work.

Error
A single consumer stops and its partition never gets processed
Root cause

A poison message makes the consumer error and crash repeatedly, stalling rebalance.

Fix

Route messages that fail N times to a DLQ with the error recorded, and let the partition continue past the bad message.

When things break

Debugging tips

  • Check kafka-consumer-groups.sh --describe for lag: which partition and consumer is behind.
  • Use a console consumer to look at raw messages and find malformed payloads (poison messages).
  • Inspect the DLQ topic for the message headers that record how many times it was retried and why.
  • Verify the commit strategy: committing before processing loses messages; committing after processing risks duplicates on crash — both are visible in the logs.
Interviews

Interview checkpoint

Practice

Hands-on lab

Publish and Consume with Kafka in Docker
45 min · intermediate
In progress
Objectives
  • Run Kafka (single-node KRaft) in Docker
  • Create a topic with multiple partitions
  • Publish messages with keys and consume them in a group
  • Measure consumer lag and replay messages by resetting offsets

Publish and Consume with Kafka in Docker

45 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/3
1

Which delivery semantic is the default for most message brokers?

2

Messages with the same key are guaranteed to be processed in order in the same Kafka partition.

3

Consumer lag is growing, one partition's consumer keeps crashing, and messages behind it are never processed. What is the most likely culprit?

Quick reference

Cheat sheet

Core terms
ProducerPublishes messages to a topic
Consumer groupWorkers that split the partitions
PartitionOrdered log unit; one consumer per partition
OffsetPosition of a consumer in a partition
LagOffset gap between consumer and latest message
DLQDead-letter queue for failed messages
kafka CLI
kafka-topics.sh --create --topic t --partitions 3Create a topic
kafka-console-producer.sh --topic tSend test messages
kafka-console-consumer.sh --topic t --group gConsume in a group
kafka-consumer-groups.sh --describe --group gShow lag per partition
kafka-topics.sh --describe --topic tShow partition layout
Go deeper

Official documentation

Apache Kafka — Introduction
docs
Apache Kafka — Design
docs
RabbitMQ Documentation
docs
Amazon SQS Developer Guide
docs