Streaming Inference
Scoring events as they flow, not as they knock
Streaming inference applies models to a continuous event stream: consume Kafka/streams, join with state, score events with bounded latency and exactly-once-ish semantics. It turns 'react now' from a request into a continuously running process.
What you will learn
- Model an inference pipeline from stream source to output
- Handle stateful joins and exactly-once semantics
- Bound latency and reprocessing on a stream
What is this?
Streaming inference consumes events (Kafka, Kinesis) and scores them continuously: deserialize, join with state/features, run the model, emit results downstream — all with bounded latency and repeatable semantics rather than a request-response loop.
Why do you need it?
- Some sources are endless streams, not request floods — logs, IoT, financial ticks.
- Stateful joins let a model see an entity's recent behavior, not just this event.
- Once-only processing keeps scoring and downstream effects consistent.
- Bounded-latency continuous runs beat cron polling for freshness.
Mental model
A baggage carousel with a trained inspector watching every bag
Streaming inference
Bags (events) never knock; they arrive continuously on the belt (topic). The inspector (model) cross-checks the bag tag against the passenger manifest (state) without stopping the belt, and only flagged bags leave the line (downstream action).
How it works
- The stream consumes events; partitions map to processing parallelism.
- A state store keeps per-key context (recent events, user profile).
- The inference function runs per event or per micro-batch with the model.
- Results emit to output topics/tables with offsets tracked.
- Checkpoints and transaction semantics give reprocessing guarantees.
A streaming inference pipeline.
In production
- A fraud stream scores every transaction within 200ms and blocks the risky ones.
- Telemetry ingestion classifies anomalies continuously back into the stream.
- Clickstream re-ranking re-scores entity recommendations as behavior arrives.
Best practices
- Key events well so state joins don't scatter across partitions.
- Make scoring deterministic per event for replay-consistent outputs.
- Set latency bounds and handle backpressure explicitly.
- Track offsets/output counts for reconciliation with the source stream.
Common mistakes
- Stateless joins producing wild results — the model can't see recent history.
- Ignoring reprocessing semantics: a restart double-scores or loses effects.
- Backpressure unmapped → the stream drains slower than input grows.
- State store unbounded → disk/memory growth forever.
Debugging tips
- 'Same event scored twice' — check delivery semantics and checkpoints across restarts.
- Stateful hits but wrong — verify partition keys; skewed keys skew the join.
- Lag grows — consumer throughput < producer rate; inspect backpressure settings.
Interview checkpoint
Hands-on lab
- Model a stream source with partitions and offsets
- Implement a stateful join per key
- Demonstrate replay-correct re-processing
Score a Stream with State
40 min · intermediate
Quick quiz
A key advantage of streaming inference over periodic batch is…
Processing semantics matter for streams: restart should resk new outputs correctly.