Cloud Data Services
Advanced 16 min read Cloud Data Services

Streaming as a Managed Service

Event streams without a 3am Kafka crash

Kinesis, Pub/Sub and Event Hubs deliver managed, durable event streams with retention, replay and autoscaling — you trade fine control for operational relief.

KinesisPub/SubEvent Hubsmanaged streamingretentionreplay
Prerequisites: Streaming Streaming Services
Before we start

What you will learn

  • Describe the core leasing of managed streaming (shards, topics, retention)
  • Choose between Kafka, Kinesis, Pub/Sub and Event Hubs for a workload
  • Design for replay and ordering within a partition/shard
Introduction

What is this?

Managed event streaming products — AWS Kinesis, Google Cloud Pub/Sub, Azure Event Hubs, plus managed Kafka (MSK, Confluent Cloud) — provide the durable-log guarantees you'd run Kafka for, but as a metered service. You configure retention, throughput and partitions; the platform handles brokers, replication, failover and autoscaling. Ordering is preserved within a shard/partition; exactly-once style guarantees are delivered per-product (e.g. ordered but at-least-once default), so you still design sinks for idempotency.

Motivation

Why do you need it?

  • No broker ops: no ZooKeeper/KRaft care, no replication tuning, no version upgrades.
  • Elastic throughput: shards/slots/publishing-quotas autoscale to your sustained rate.
  • Built-in retention + replay from the console rather than custom tooling.
  • Native integration with the vendor's compute (Lambda, Dataflow/BigQuery, Functions/Synapse).
Think about it

Mental model

Imagine

A bank's armored courier service vs mailing letters yourself

Think of

Managed streaming vs self-hosted Kafka

The courier guarantees the route, insurance and priority lanes so you can ship envelopes without running the trucks. Managed streams guarantee delivery, ordering and replay while you never touch a broker.

Mechanics

How it works

  • Kinesis: data streams split into shards (each 1MB/s in, 2MB/s out READ limits); consumers read into Lambda/Kinesis Data Analytics/Flink.
  • Pub/Sub: topics with pull/push subscribers; ordering only within message ordering keys; big retention window; flow control via ack/dead-letter.
  • Event Hubs: partition-key-based ordering with consumer groups and checkpointing to a storage account.
  • Managed Kafka (MSK/Confluent): same client protocol you host Kafka uses, minus the ops.
  • All emit per-stream metrics (in/out bytes, lag, age of last record) you can alert on.

Managed streaming anatomy: producers to shards to consumers.

Producers
Shards
Retention
Consumers
Metrics
Real world

In production

  • A real-time dashboard ingests user events into Kinesis, Lambda enriches them, and At-SQL aggregates windows — all without a single broker on the call.
  • A retailer sends order confirmations via Cloud Pub/Sub with a 7-day retention window for replay when a fulfillment system hiccups.
  • An Azure org routes IoT telemetry into Event Hubs with checkpointed consumers recovering to the last committed offset after a deploy.
Guidelines

Best practices

  • Design sinks to be idempotent — managed streams default at-least-once; exactly-once is a product wrinkle, not a given.
  • Partition by a meaningful key (user/tenant) when per-entity ordering matters.
  • Set retention deliberately: replay needs are time-bounded; big retention = storage cost at scale.
  • Alert on consumer lag and 'age of last record' — a healthy broker with a dead consumer is still an outage.
Watch out

Common mistakes

  • Assuming Pub/Sub/Kinesis give Kafka-style global ordering — only per-shard/partition ordering is guaranteed.
  • Choosing a service by headline only: MSK keeps Kafka semantics but adds ops; Kinesis cuts ops but changes your client protocold.
  • Forgetting that at-least-once means every sink needs a dedupe key.
  • Letting autoscaling be magic: unbounded in-flight unconsumed data is still a bill and a backlog.
When things break

Debugging tips

  • When events are missing, check for just-out-of-license retention expiry first, then consumer lag then producer metadata.
  • Ordering bugs → confirm the producer used the partition key (not a random hash) and a single partition per entity.
  • Lag grows → consumer autoscale can't keep up; add consumers or re-summarize at the edge.
  • Duplicate events → standard at-least-once semantics; dedupe by payload key in the sink, not by re-cranking retries.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

You need per-entity ordering across millions of entities. The correct setup is:

2

Managed streams guarantee at-most-once delivery by default.

Go deeper

Official documentation

Kinesis Data Streams
docs
Cloud Pub/Sub overview
docs