Streaming
Intermediate 16 min read Streaming

Kafka Topics and Partitions

The durable, partitioned append-only log

A Kafka topic is an append-only, partitioned log. Partition count decides parallelism and ordering scope; replication guarantees durability. Designing topics (keys, count, retention) is a core data-engineering skill.

kafkatopicpartitionreplicationkeyretentionlogordering
Prerequisites: Streaming Fundamentals
Before we start

What you will learn

  • Explain topic → partition → replica structure
  • Design partition counts and keys for parallelism + ordering
  • Set retention and replication to match SLAs and cost
Introduction

What is this?

A Kafka topic is a named, immutable, append-only log split into partitions, each partition being an ordered sequence of records. Producers choose a partition (via a key or round-robin), each partition is replicated to multiple brokers for durability, and consumers divide partitions among them. Topic design — keying, partition count, retention, replication — determines throughput, ordering scope and recovery capability.

Motivation

Why do you need it?

  • Partitions are the unit of parallelism for both producers and consumers.
  • Ordering is guaranteed within a partition — so keys that must stay ordered share a partition.
  • Replication copies each partition across brokers so broker failure loses nothing committed.
  • Retention governs replay depth vs storage cost — the core trade for event-driven systems.
Think about it

Mental model

Imagine

A filing cabinet with labeled drawers

Think of

Topic and partitions

The topic is the cabinet; each drawer a partition holding an ordered stack of records. If a duplicate subject must stay in order, it goes in the same drawer (same key). Each clerk (consumer) reads drawers in sequence without mixing them.

Mechanics

How it works

  • Producers append records to topics; the broker assigns each to a partition.
  • With a key, the record's partition = hash(key) % partitions — same key, same partition, ordering preserved.
  • Without a key, round-robin spreads records for higher aggregate throughput.
  • Each partition has replicas on different brokers; the leader serves reads/writes, followers stay in sync.
  • Retention (time-large, size-large, log compaction) auto-deletes or compacts old records.

Topic layout across partitions and replicas.

Topic: orders
Partition 0
Partition 1
Replicas
Consumers
Real world

In production

  • Click events keyed by session_id preserve per-session order while parallelizing across sessions.
  • A 48-partition topic matches the consumer group's parallelism on a 48-executor cluster.
  • Ride-hailing orders replicated 3x survive the loss of any single broker.
Guidelines

Best practices

  • Dimension partitions to your consumer parallelism AND peak throughput, not to 'some number'.
  • Key related events with a stable, bounded-cardinality key (user_id, device_id) for ordered processing.
  • Control retention deliberately: days of data you might actually replay, not 'forever'.
  • Set short- to-medium retention with replication for the dual guarantee of durability + replay.
Watch out

Common mistakes

  • Hundreds of partitions but a consumer group with 4 instances leaves 96 idle.
  • Choosing a huge-cardinality or unbounded key that produces a partition per garbage value.
  • Turning up replication to survive a failure and paying 3x storage you never use.
  • Putting 10MB blobs on a 'stream' topic, turning the broker into a file server.
Root cause analysis

Common issues

Error
One partition always has huge lag, others zero
Root cause

A hot key or non-uniform keying overloads one partition

Fix

Salt the hot key or widen the key space so partitions balance.

Error
Committed data is lost when a broker fails
Root cause

Replication factor 1 or an unclean leader election losing un-replicated + acked data

Fix

Set replication factor ≥3 and wait for ISR acks; verify min.insync.replicas.

Error
Replay window too short to rebuild state
Root cause

Retention too small (time.minutes/size.max)

Fix

Grow retention to match your recovery SLA; use compacted topics for keyed state.

When things break

Debugging tips

  • Check partition vs consumer-group size for idle partitions (underused parallelism).
  • Lag per partition tells you the slowest consumer; balance hot partitions by keying.
  • Message ordering issue? Verify the key is stable — a changing key breaks the same-partition path.
  • Use kafka-topics to inspect partitions and kafka-consumer-groups for lag distribution.
Interviews

Interview checkpoint

Practice

Hands-on lab

Design and Inspect Kafka Topics
40 min · intermediate
In progress
Objectives
  • Create topics with chosen partitions and replication
  • Publish keyed events and inspect partition assignment
  • Measure retention/replication settings and behavior

Design and Inspect Kafka Topics

40 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Two events for the same user arrive at different partitions. What is lost?

2

Kafka topics delete records immediately once all consumers have read them.

Go deeper

Official documentation

Kafka — topics & logs
docs
Kafka — ISR/replication
docs