Streaming
Advanced 17 min read Streaming

Kafka Operations and Performance

Running the log well and knowing when it hurts

Operating Kafka means knowing the broker-side metrics that matter: ISR health, request rate, replicas-in-sync, disk/IO, and unclean leader election. Tuning spans batch sizes, partition count, and replication for the workload.

kafka opsISRreplicationunclean electionmetricsthroughputlurker
Prerequisites: Backpressure and Throttling
Before we start

What you will learn

  • Read the key broker metrics: ISR, leader, request latency, disk I/O
  • Explain unclean leader election and why you may disable it
  • Tune replication, partitions and batching for throughput
Introduction

What is this?

Kafka operations is the discipline of monitoring broker health (ISR in-sync replicas, leader distribution, request latency), replication behavior (min.insync.replicas, unclean election), and resource constraints (disk I/O, page cache, GC). Performance tuning centers on producer batching, partition count, replication factor and storage layout — knowing each knob's cost and effect.

Motivation

Why do you need it?

  • A healthy cluster quietly serves the platform; a misconfigured one loses data at the worst moments.
  • ISR/unclean-election settings decide whether you lose acknowledged data.
  • Disk I/O and page cache dominate Kafka throughput more than CPU — tuning focuses there.
  • ops. metrics are the shared language with platform/on-call teams.
Think about it

Mental model

Imagine

A fleet maintenance crew

Think of

Kafka operations

You don't wait for planes to crash: you check oil, tire pressure, fuel levels per plane (ISR health, request rate, disk), rotate planes to even wear (leader balance), and keep spares (replicas) tuned so a single failure never grounds the airline.

Mechanics

How it works

  • Each partition has a leader and follower replicas; ISR = replicas caught up enough to take over.
  • Writes go to the leader and are replicated to ISR followers; acks=all waits for ISR ack.
  • If all ISR members die, a broker may elect a leader from out-of-sync replicas (unclean) — data loss risk.
  • Throughput = aggregate of partitions' leaders × per-partition write path; batching + compression lift per-partition rate.
  • Broker metrics: UnderReplicatedPartitions, RequestQueueTime, ISR shrink/grow, disk utilization, GC.

Replication, ISR and the metrics loop.

Leader
Followers
Ack (acks=all)
Metrics
Alert
Real world

In production

  • min.insync.replicas=2 + acks=all means a broker glitch can't silently drop acked records.
  • unclean.leader.election=false protects the system-of-record from data loss in exchange for availability.
  • Partition rebalance tooling moves leaders to spread load after a broker restart.
Guidelines

Best practices

  • Set min.insync.replicas and acks=all together on critical topics.
  • Watch UnderReplicatedPartitions and ISR-size drops as primary health signals.
  • Keep partitions-to-topic moderate; thousands of partitions waste memory and metadata.
  • Prefer an even leader distribution and monitor per-broker request latency.
Watch out

Common mistakes

  • acks=all with min.insync.replicas=1 — effectively acks=1, no real durability.
  • Unclean election left enabled on the financial topic 'for availability'.
  • Underpartitioning so one leader handles everything; overpartitioning so metadata churns.
  • Storage on spinning disks for a high-throughput log — page cache starvation.
Root cause analysis

Common issues

Error
Leader-less partition after two brokers died
Root cause

Unclean election disabled, and the remaining replicas are out-of-sync

Fix

Restore a broker or, if you accept loss, flip unclean election on briefly with a plan.

Error
Write latency spikes under load
Root cause

Disk/IO saturated or acks=all followers slow

Fix

Add partitions to spread load; check follower fetch and disk throughput.

Error
Topic grows unreachable (no leader)
Root cause

Replication factor too low / ISR empty after a rolling restart

Fix

Ensure RF≥3, min.insync.replicas≥2, rolling restarts sequential; use controller leader moves.

When things break

Debugging tips

  • UnderReplicatedPartitions persists → ISR members lost or slow; look at follower fetch time.
  • High request time paired with queue time → broker capacity or GC pressure.
  • Spike to zero requests for a partition = leaders for it gone; check election state.
  • IOUtilization high → increase partitions (more disks in flight) or reduce batching of large payloads.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The immediate risk of acks=all with min.insync.replicas=1 is...

2

Enabling unclean leader election always preserves data consistency.

Go deeper

Official documentation

Kafka configuration reference
docs
Confluent Kafka metrics
docs