Apache Flink Basics
Streaming as a first-class citizen
Flink is a true event-at-a-time stream processor: stateful operators, checkpointing for exactly-once, and out-of-order event-time handling built in. It's the engine teams pick when sub-second latency plus state matter.
What you will learn
- Explain how Flink differs from the micro-batch model
- Describe stateful operators and checkpointing
- Understand windows and event-time semantics
What is this?
Apache Flink is a distributed stream-processing engine with continuous, event-at-a-time execution. It manages per-key state, supports event-time windows with watermarks and allowed-lateness, and achieves exactly-once through periodic checkpointing of state plus a transactional sink handshake. Where batch engines periodically re-query an unbounded input, Flink is designed to be the streaming state machine itself.
Why do you need it?
- Sub-second latencies with true event-time semantics for out-of-order data.
- Stateful operators (session windows, CEP, custom state) out of the box.
- Exactly-once via checkpoints — recovery without duplicates.
- It remains the reference point for anything the micro-batch model can't do.
Mental model
A live TV editor
Flink processing per event
The editor decides frame-by-frame, keeping running state about who's on camera (stateful), and can rewind to any checkpoint to resume a broadcast cleanly (checkpointing) — even when footage arrives out of order (event time).
How it works
- Jobs are dataflow graphs of operators: sources, transforms, windows, joins, sinks.
- Operators run per-parallelism task; state is keyed and stored locally with RocksDB or heap.
- Watermarks carry event-time progress; late events up to allowed-lateness update open windows.
- Checkpoint barriers flow through the topology; on the barrier, state snapshots are persisted.
- Exactly-once sinks converge: checkpoints + transactional two-phase commit between Flink and the sink.
Flink dataflow with state and checkpoints.
In production
- A ride-hailing platform computes per-driver session windows upstream, streaming to dashboards.
- Real-time fraud runs keyed CEP patterns on card-swipe streams with sub-second windows.
- Flink backs materialized views that update the moment source rows arrive.
Best practices
- Key by the entity whose state matters (user, device) for most operations.
- Use event time + watermark + allowed-lateness instead of processing-time windows.
- Cap state via TTL/retention or state grows unbounded and eats the checkpoint budget.
- Run the checkpoint interval and alignment tuned to your SLA (not the default).
Common mistakes
- Using processing time windows while claiming event-time correctness.
- Keying a global rollup on a dummy key — that's a single-node bottleneck masquerading as parallel.
- Ignoring state TTL, then watching checkpoints balloon.
- Applying checkpointing to a stateless op and calling it 'exactly-once' (sinks must commit).
Common issues
Output has duplicates after recoverySink isn't transactional/exactly-once-compatible; checkpoint restored but sink replayed
Use the appropriate exactly-once Kafka/committing sink or dedupe with an idempotency key.
Checkpoints fail constantlyState too large for the interval, or a slow operator stalls barriers
Raise checkpoint interval, add state TTL, or increase checkpointer resources.
Job is 90% idle operatorsParallelism mismatch: sources at 20, downstream at 2
Match the parallelism and rebalance keying to distribute work.
Debugging tips
- Late events? Inspect the actual event-time vs watermark for the topic.
- Restart storms often mean checkpoint misalignment or a failing sink handshake.
- State growth → check TTL config and whether a key is hot (cardinality).
- Look for backpressure in the UI (green→red) before blaming sources.
Interview checkpoint
Quick quiz
Which is Flink's key storage that scales to big state?
With Flink, replay isn't possible after the stream advances beyond a window.