Streaming
Intermediate 15 min read Streaming

Consumer Groups and Rebalancing

Parallel consumers with safe partition ownership

A consumer group divides a topic's partitions among its members so each partition is processed by exactly one consumer at a time. Rebalancing reassigns partitions when members join or leave — and mishandled, it causes storms and duplicates.

consumer grouppartition assignmentrebalancingcoordinatorlagstatic membership
Prerequisites: Kafka Producers and Consumers
Before we start

What you will learn

  • Explain partition assignment rules for a consumer group
  • Describe the rebalance lifecycle and its failure modes
  • Tune rebalance and avoid storms
Introduction

What is this?

A consumer group is a set of consumers that collectively read a topic, splitting its partitions among themselves so each partition is owned by one member at a time. A group coordinator supervises membership and assignment; when a member joins, leaves or fails, the group rebalances — revoking and reassigning partitions. Rebalances are normal but costly: they pause all members and can loop into 'rebalance storms' if config is wrong.

Motivation

Why do you need it?

  • Groups let you scale consumers: adding consumers adds partition-assignment parallelism, up to the partition count.
  • Per-partition single-owner semantics keep processing deterministic and state simple.
  • Rebalances enable rolling deploys and crash recovery automatically.
  • Understanding rebalances explains lag spikes, duplicates and 'members keep disappearing' bugs.
Think about it

Mental model

Imagine

Assigning tables to waiters

Think of

Consumer groups

Each waiter (consumer) takes a set of tables (partitions). If a waiter goes home, a manager (coordinator) re-assigns their tables — and it's noisy: everyone pauses while the new plan is announced (rebalance).

Mechanics

How it works

  • Consumers in a group greet the coordinator and request membership.
  • The leader picks an assignment strategy (range, round-robin, sticky) that maps partitions to members.
  • Members fetch the assignment; each owns a disjoint set of partitions and consumes them.
  • On membership change, the coordinator triggers a rebalance: members revoke partitions, rejoin, and get a new assignment.
  • With eager rebalancing the whole group pauses; incremental/sticky rebalancing reassigns minimally.

A consumer group rebalancing.

Group Coordinator
Members
Assignment
Rebalance
Stable state
Real world

In production

  • A consumer group with 6 instances reads a 12-partition topic two-at-a-time.
  • A rolling deploy restarts consumers one by one; sticky rebalancing keeps most partitions untouched.
  • A failed consumer's partitions are reassigned to remaining members in seconds (session timeout).
Guidelines

Best practices

  • Right-size the group: roughly partitions = members for linear scaling (more members than partitions is idle).
  • Prefer sticky/incremental rebalancing to reduce churn on short-lived changes.
  • Set session.timeout.ms and max.poll.interval.ms together — a mismatched pair causes kick-outs.
  • For stateful consumers, use static membership (group.instance.id) so restarts don't trigger full rebalances.
Watch out

Common mistakes

  • More consumers than partitions — half the group idles forever.
  • max.poll.interval.ms too low → processing takes longer, members time out, infinite rebalance loop.
  • No group.instance.id for stateful workers → every restart wipes local state ownership.
  • Unbounded poll() heartbeat costs under a heavy scheduler, triggering session timeouts.
Root cause analysis

Common issues

Error
Infinite rebalance loop after deploy
Root cause

Consumers join, then get kicked (session timeout), rejoin, repeat

Fix

Raise session.timeout.ms / max.poll.interval.ms; check slow GC or heavy startup.

Error
Duplicate processing after a rebalance
Root cause

Partition ownership changed while offsets weren't committed

Fix

Commit offsets timely (after batch) or use transactions; enable boundary idempotence.

Error
Adding members makes processing slower
Root cause

Rebalance churn (every join triggers a global pause) with more partitions than needed

Fix

Cap members ≈ partitions; enable incremental rebalancing; use static membership.

When things break

Debugging tips

  • Repeated 'consumer rebalance' logs = loop; check poll interval vs processing time.
  • Lag on some partitions but not others → assignment imbalance, check sticky prefers spread.
  • See 'member … left unexpectedly' → sessions timing out; correlate with GC/pause logs.
  • Describe the group (output with active members and current offsets) — it's the first tool.
Interviews

Interview checkpoint

Practice

Hands-on lab

Observe Rebalances and Lag on a Live Group
35 min · intermediate
In progress
Objectives
  • Run a two-consumer group on a two-partition topic
  • Trigger rebalances by adding/removing a consumer
  • Measure lag and assignment with describe

Observe Rebalances and Lag on a Live Group

35 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A group of 4 consumes a topic with 3 partitions. How many members process data concurrently?

2

Rebalancing always pauses the entire consumer group.

Go deeper

Official documentation

Kafka consumers & groups
docs
Kafka rebalance protocols
docs