Streaming
Intermediate 16 min read Streaming

Kafka Producers and Consumers

Tuning the two ends of the pipe

Producers batch, compress, key and wait for acks; consumers pull in batches, process, and commit offsets. Ack levels, idempotence, and batch tuning dominate producer performance; commit timing and batching dominate consumer behavior.

kafkaproducerconsumeracksbatchcompressionoffset commitidempotence
Prerequisites: Kafka Topics and Partitions
Before we start

What you will learn

  • Configure producer acks, batching and idempotence
  • Explain consumer batch processing and offset commit strategies
  • Diagnose lag from producer/consumer mis-configuration
Introduction

What is this?

A Kafka producer turns business events into batches of records, keyed and compressed, delivered to a broker with a chosen ack level (at-most, at-least, or exactly via idempotence). A consumer pulls batches, processes them, and either auto- or manually commits offsets. Most real-world Kafka problems are producer/consumer configuration — batch size, linger, acks, commit interval.

Motivation

Why do you need it?

  • Ack level is a durability vs latency dial: acks=all for finance, acks=0 for telemetry.
  • Batching + compression can multiply throughput 10x for small, high-cardinality events.
  • Manual offset commits are how you avoid processing/skip races.
  • End-to-end, producer and consumer configs set the pipeline's latency and cost.
Think about it

Mental model

Imagine

Sending packages at a parcel depot

Think of

Producer batching

The depot stacks outgoing parcels into a truck (batch), seals it (compression), and gets a signature upon delivery (ack). Linger = how long the depot waits to fill a truck instead of sending one parcel at a time.

Mechanics

How it works

  • Producers buffer records per topic-partition, then send batches when the size threshold or linger.ms elapses.
  • acks=0 fire and forget; acks=1 leader committed; acks=all (with min.insync.replicas) full durability.
  • enable.idempotence=true makes the producer retry-safe: it tags each batch with a sequence the broker dedupes.
  • Consumers poll() in batches (max.poll.records), process, then commit offsets (auto or manual).
  • Commit before processing risks loss; commit after risks reprocessing duplicates — it's a trade.

Producer batching and consumer offset commit.

Producer buffer
Send batch
Broker append
Consumer poll
Commit offset
Real world

In production

  • A payment producer runs acks=all + idempotence so no charge event is ever lost or duplicated.
  • A telemetry producer uses acks=0 for speed, accepting rare loss of non-critical metrics.
  • A consumer processes in batches then commits once per batch — crash reprocesses a window, never skips.
Guidelines

Best practices

  • Batch aggressively for small events: 16-64KB or a few dozen ms of linger pay for themselves.
  • Use acks=all + idempotence for anything that matters; reserve acks=1/0 for loss-tolerant telemetry.
  • Compress (lz4/snappy/zstd) — high-cardinality JSON payloads shrink dramatically.
  • Commit offsets after processing, and make processing idempotent so replays are safe.
Watch out

Common mistakes

  • sending one record at a time with no batching — the classic 'misconfigured producer' lag.
  • acks=all with no compression, trading away two free wins.
  • Auto-commit with slow processing loops — offsets jump past unprocessed batches.
  • Consumer poll loops with huge max.poll.records and long processing — session timeout evictions.
Root cause analysis

Common issues

Error
Orders topic lag despite idle consuming group
Root cause

Producer sends 1-record batches (tiny batch.size); broker/serialization is the bottleneck

Fix

Raise batch.size and linger.ms; enable compression.

Error
Consumers kicked out mid-job (members leaving)
Root cause

Processing exceeds max.poll.interval.ms

Fix

Shorten batches or raise max.poll.interval.ms/MaxXX; process concurrently.

Error
Duplicates on the sink after every restart
Root cause

commit before process on crash, or producer retries without idempotence

Fix

Commit after successful processing and enable producer idempotence + idempotent sink.

When things break

Debugging tips

  • Throughput flat? Check linger.ms/batch.size — a producer without batching is the usual culprit.
  • Duplicate processing check → look at commit strategy vs crash-replay behavior.
  • Consumer getting evicted (Rebalance) — raise max.poll.interval.ms or shrink batch size.
  • Latency spikes → check broker-side metrics, but confirm producer acks=all vs=0 first.
Interviews

Interview checkpoint

Practice

Hands-on lab

Tune a Python Producer and Consumer
35 min · intermediate
In progress
Objectives
  • Run a batching producer and measure send throughput
  • Compare acks and linger impact
  • Run a consumer with manual offset commits

Tune a Python Producer and Consumer

35 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A pipeline has tiny events and terrible throughput. The most likely producer fix is...

2

Auto-commit is safe for consumers with long processing windows.

Go deeper

Official documentation

Kafka producer configs
docs
Kafka consumer configs
docs