Streaming
Advanced 17 min read Streaming

Delivery Semantics and Exactly-Once

What 'once' even means across a distributed pipeline

At-most-once, at-least-once and exactly-once describe how source retries and sink idempotency combine. Exactly-once in streams requires the trinity: transactional producer, atomic state, and idempotent sinks.

delivery semanticsat-least-onceat-most-onceexactly-onceidempotenttransactional produceroffset
Prerequisites: Kafka Producers and Consumers
Before we start

What you will learn

  • Define the three delivery levels precisely
  • Explain why exactly-once needs producer + state + sink to cooperate
  • Decide the right semantics for a workload
Introduction

What is this?

Delivery semantics describe what happens to a record when retries occur. At-most-once never repeats but can lose. At-least-once never loses but can repeat. Exactly-once appears to process each record once — end to end — which in practice comes from *idempotent* sinks plus coordinated offsets and state. Kafka's transactional producer and Flink's checkpointing are the reference implementations.

Motivation

Why do you need it?

  • Finance, payments and inventory need 'no loss, no double count' — exactly the hardest guarantee.
  • Most analytics tolerate at-least-once provided downstream dedupes.
  • Misreading 'exactly-once' as free leads to features that silently duplicate funds.
  • Knowing the mechanics lets you state guarantees users can actually rely on.
Think about it

Mental model

Imagine

Cooking that never re-fires a dish

Think of

Delivery semantics in kitchen terms

At-most-once: the dish is sent once, but a messed-up plate is thrown away, not replaced (loss possible). At-least-once: if anything goes wrong you always cook again (duplicates possible). Exactly-once: choose by order number, so re-cooking produces the identical dish, safe to serve twice (idempotency).

Mechanics

How it works

  • At-most-once: producer gets acks=0/1, no retries — a record may fail silently once.
  • At-least-once: producer retries and consumers commit *after* processing — duplicates possible on retry/rebalance.
  • Idempotency: the sink dedupes by key ($.key_id, hash of content) so repeated writes collapse.
  • Transactional producer: batches + offsets + state commit atomically through Kafka transactions.
  • Exactly-once end-to-end = transactions (source→Kafka) + checkpointed state + idempotent sink (Kafka→sink).

Layers needed for end-to-end exactly-once.

Source
Kafka broker
Processor
Sink
Consumers
Real world

In production

  • A payments engine consumes transactions with exactly-once so a retried packet never double-charges a card.
  • Telemetry dashboards run at-least-once; metric sinks use keyed dedupe to absorb duplicates.
  • A library-booking system uses at-most-once for cache invalidation alerts (missing an alert is fine).
Guidelines

Best practices

  • Start from the business need: payments need EOS; analytics almost never do.
  • Prefer idempotent sinks + at-least-once for the 90% case — far cheaper than full EOS.
  • Only buy EOS where loss AND duplication are both unacceptable.
  • When using EOS, read_committed consumers and disabled read_uncommitted for the pipeline.
Watch out

Common mistakes

  • Claiming exactly-once on a pipeline whose sink duplicates (the sink wasn't part of the guarantee).
  • Using idempotency keys that are too coarse, collapsing legitimately identical events.
  • Relying on exactly-once 'for free' from a broker while source processing replays state twice.
  • Paying EOS cost for a workload at-least-once would satisfy.
Root cause analysis

Common issues

Error
Exactly-once configured but money is still double-counted
Root cause

The sink isn't idempotent — EOS only covers producer→consumer so far

Fix

Make the sink upsert by unique key or participate in the transaction boundary.

Error
Duplicates after consumer rebalance despite EOS
Root cause

Consumers not reading from the read_committed snapshot, or state not restored atomically

Fix

Set isolation.level=read_committed and restore from the checkpoint.

Error
EOS setup 4x slower
Root cause

Transactions and coordination add latency/overhead for a workload that didn't need it

Fix

Re-validate the SLA; downgrade to at-least-once + idempotent sink.

When things break

Debugging tips

  • See duplicates? First identify WHERE they're created: producer retry, consumer rebalance, or sink non-idempotency.
  • Verify the sink: does it upsert by primary key? — that's the whole game.
  • Read committed vs read_uncommitted consumers — if EOS expected, wrong isolation causes phantom duplicates.
  • Look at offset commits vs processing: duplicates on commit-after semantics are expected; check where.
Interviews

Interview checkpoint

Practice

Hands-on lab

Make an At-Least-Once Consumer Idempotent
35 min · advanced
In progress
Objectives
  • Prove at-least-once replays create duplicates in a naive sink
  • Build an idempotent sink keyed by event_id and watch duplicates collapse
  • Explain when exactly-once machinery is needed vs when idempotency suffices

Make an At-Least-Once Consumer Idempotent

35 min · advanced

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A retried broker write double-appends to a sink with no unique key. Which guarantee is broken?

2

At-least-once guarantees every record is delivered at least once.

Go deeper

Official documentation

Kafka semantics (exactly-once)
docs
Flink exactly-once checkpointing
docs