Backpressure and Throttling
The feedback signals of a pipeline that can't keep up
Backpressure is the pipeline's way of telling an upstream it can't keep up: Kafka pushes it into lag, Flink signals it with buffer pressures, and you tune with consumers, partitions, or application-level throttling.
What you will learn
- Identify backpressure and where it propagates
- Explain the distinction between broker lag and engine-side backpressure
- Apply throttling, partitioning and parallelism remediations
What is this?
Backpressure is what happens when a downstream stage can't consume as fast as it's being fed. In Kafka, it shows as consumer lag (offsets behind). In engines like Flink, operators send backpressure signals upstream as buffers fill, which throttles producers or sheds load. Throttling is the deliberate rate-limiting of production or consumption to protect a system. Managing both is how you keep pipelines stable under bursts.
Why do you need it?
- Unchecked lag turns a 'real-time' system into a delayed one with silent staleness.
- Applying backpressure at the source beats letting a fan-in bottleneck crash.
- Bursty sources happen; a well-tuned pipeline breathes with them instead of breaking.
- Understanding backpressure explains why more partitions don't always fix throughput.
Mental model
A clogged airport security lane
Backpressure
When the scanner is slower than the arrivals, the queue (lag) lengthens. Agents signal upstream to slow the flow (backpressure), or redirect to more lanes (parallelism). Squeezing more arrivals only makes the queue longer.
How it works
- Broker-side: producers append at rate P; consumers read at rate C; unread records accumulate as lag.
- Consumer-side: the consumer's polling, processing and committing rate defines C; if C < P, lag grows.
- Engine-side: Flink buffers between tasks; when downstream is slow, buffers fill and upstream tasks are throttled and down-tuned.
- Application throttling caps P (rate limiter) so you never exceed system capacity on purpose.
- Remedying: add consumer instances/partitions, increase parallelism, or reduce per-record cost.
Where backpressure signals appear.
In production
- A consumer group lag monitor pages when lag > threshold, before data skews the dashboard.
- Flink operators display red 'backpressure' indicators when a slow sink stalls the graph.
- A producer limiter allows 5k events/s so the broker never OOMs on a burst day.
Best practices
- Monitor consumer lag and set lag alerts with recovery time, not just existence.
- Match consumers to partitions; scale the whole parallel path together.
- Prefer 'slow down upstream politely' (Flink) to random backends repeated retries.
- Throttle bursty producers at the boundary with permitted-rate budgets per tenant.
Common mistakes
- Adding consumers beyond partition count — the extra members do nothing for lag.
- Rate-limiting the broker instead of the producer, turning a burst into a stall.
- Ignoring sink backpressure (a slow ES/DWH sink throttling the whole topic).
- Auto-scaling consumers without scaling partitions — pointless elasticity.
Common issues
Dashboard lag grows dailyCDC/long-batch sink slower than producer; no throttling
Throttle producer at peak, add parallelism, or switch to bounded sink batching.
Flink graph stalls at one operatorA resource-bound operator (serialization or sink) buffers full
Increase that operator's parallelism; add backpressure-aware sink buffering.
Producer OOM on bursty write spikesUnbounded in-flight requests (no max.in.flight / no rate guard)
Set producer buffer limits + linger, or rate-limit at the API boundary.
Debugging tips
- Lag high but consumers idle → partitions under-assigned; check assignment.
- Flink shows red backpressure but Kafka lag is 0 → the engine is the bottleneck, not the broker.
- Compare consumer poll rate to producer rate; ratio defines your runway.
- P90 consumer processing time × events per poll = the sink budget reality.
Interview checkpoint
Hands-on lab
- Generate a producer burst faster than a consumer can keep up
- Measure consumer lag with kafka-consumer-groups
- Throttle or parallelize to drain the lag and confirm recovery
Produce a Burst, Then Watch Lag Drain
30 min · advanced
Quick quiz
Lag remains high even though you added consumer instances. The most likely problem is...
Backpressure is always a sign of a config error.