Streaming
Advanced 16 min read Streaming

Streaming Data Quality

Checking quality without the luxury of a re-run

Data quality in streaming is harder than in batch because there's no 're-run': expectations, anomaly detection and out-of-band correction must run inline, with lag as a first-class metric alongside freshness.

streaming data qualityexpectationsanomalydriftlate dataquality gates
Prerequisites: Stream Processing Basics
Before we start

What you will learn

  • Apply expectations inline on streaming events
  • Detect anomalies and drift with windowed metrics
  • Build correction paths for bad and late data
Introduction

What is this?

Streaming quality checks run inline: each event (or batch) is validated against expectations (schema, nulls, ranges, value distributions), anomalies are detected by comparing windowed metrics, and bad events are routed to a dead-letter or correction lane. Because streams can't be 're-run yesterday', the quality of each arrival window matters as much as the final result.

Motivation

Why do you need it?

  • Batch QA flows (re-run, fix, redeploy) don't exist; a broken producer emits bad events for days.
  • Stream consumers (alerts, live dashboards) act on first pass — quality must gate there.
  • Lag, duplication and ordering IS data quality for streams — not just value correctness.
  • Drift detection catches slow degradation before users see wrong numbers.
Think about it

Mental model

Imagine

An airport security scanner per passenger

Think of

Inline stream checks

You can't re-screen the whole terminal later — each passenger is inspected as they arrive (schema/range check), flagged anomalies alert the team immediately (anomaly detection), and the false positives go to a slow lane (dead letter) for retreading later.

Mechanics

How it works

  • Expectation checks run per-event/per-window: non-null, type, allowed values, range, cardinality, referential.
  • Anomaly detection compares windowed rates (events/sec, null-rate) to rolling baselines (z-scores, EMA).
  • Bad events route to a DLQ (topic) with the reason; valid events continue on the main path.
  • Late-data and skew handling feeds corrections back (allowedLateness + late stream).
  • Quality metrics themselves are streamed (dashboards for error-rate, lag, drift) and alerted on.

Inline quality gates in the stream.

Events
Inline checks
Anomaly engine
DLQ
Clean path
Real world

In production

  • A clickstream pipeline drops events with user_id null to a DLQ and alerts on the null-rate gate.
  • An order stream detects a 30% drop in event volume vs the rolling baseline, paging before the dashboard lies.
  • A metrics event stream re-synced with allowedLateness and a late-data lane for exact reporting.
Guidelines

Best practices

  • Check events as they arrive and treat the DLQ as a first-class consumer (not 'a folder').
  • Alert on quality rates, not absolute counts — a friendly baseline prevents noise.
  • Encode quality metrics in the stream itself for self-observability.
  • Design for latency: inline matching should run at wire speed, not asynchronously after deadlines.
Watch out

Common mistakes

  • Letting 'quality' run only in nightly batch — by then worst damage is done.
  • Using a DLQ nobody owns — bad events sit there forever, invisible.
  • Alerting on absolute numbers that spike on every traffic burst (false alarms burn focus).
  • Checking only value correctness but not ordering/duplication/lag quality.
Root cause analysis

Common issues

Error
Alerts fire every traffic spike
Root cause

Absolute thresholds instead of baseline-relative anomaly detection

Fix

Use rolling-window baseline + z-score/EMA so dots scale with normal variance.

Error
Bad events keep flowing downstream past your check
Root cause

Check ordering: schema registry passes but value validation runs too late

Fix

Honor quality gates inline before branching; test the DLQ path end to end.

Error
Metrics drift on late events
Root cause

Late arrivals reprocessing closed windows without corrections

Fix

allowedLateness + a late-data lane; re-emit corrected facts with watermark timestamps.

When things break

Debugging tips

  • A metric dropping suddenly → compare production rate to baseline; sudden dip = producer or topology issue.
  • DLQ filling → check the newest schema change or a source-side change (types, null policy).
  • Late data suddenly arriving → check consumer lag and scheduling (the quality metric 'lag').
  • Suspicious ordering → keying change or partition rebalance; verify keyBy stability.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The best first stop for an event that fails schema or value checks in a stream is...

2

Lag is a data-quality metric for streaming, not just an ops metric.

Go deeper

Official documentation

Great Expectations streaming
docs
Flink quality-at-speed docs
docs