Validation in Batch and Streaming

• Position validation gates in a batch pipeline (schema, per-row, aggregate)
• Route invalid streaming records to dead-letter lanes instead of dropping them
• Compare cost/latency trade-offs of blocking vs quarantine strategies

• Catching garbage at the boundary avoids corrupting modeled tables you can't easily repair.
• Dead-lettering keeps the stream moving so good events aren't delayed by bad ones.
• Validation at the right stage (source vs target) determines whether you fix data or fix the pipeline.
• Streaming validation matters early: memory/state utopias reconstruct nothing if bad data is already aggregated.

Security doesn't halt the airport when one bag looks odd — it routes that bag to a quarantine lane while the rest of the flight proceeds. Batch validation is more like a customs check that stops the whole truck before the border.

• Batch gate 1 (landing): schema + volume checks on raw copy; failure stops downstream.
• Batch gate 2 (transform): grain and metric assertions after each model build.
• Batch gate 3 (publish): final expectations before the table is made available to consumers.
• Stream per-record: schema/domain rules evaluate each event; invalid ones go to a dead-letter topic with the original payload + reason.
• Stream windowed: per-window aggregates compared to baselines; anomalies emit alerts and trigger recompute.
• Reconciliation between validated lanes and downstream consumers verifies nothing was silently dropped.

• An e-commerce company routes malformed click events to a dead-letter topic and replays them after fixing the producer.
• A batch insurance loader fails landing when the daily premium file count is outside ±5%, before any model runs.
• A gaming analytics stream computes per-window average playtime and alerts when it drops below a baseline — flagging a telemetry outage before dashboards show zeros.

• Keep the original payload in the dead-letter record with a machine-readable reason code.
• Make validation rules versioned and shared across batch and stream paths.
• Sample-and-fail for aggregates, but hard-fail for schema-breaking or PII-missing violations.
• Instrument validation outcomes: counts validated, quarantined, rejected per gate per run.

• Silently dropping invalid streaming records — no dead letter, no replay, no audit.
• Only validating the final table, so errors were already baked into intermediate state.
• Applying streaming per-record rules so strict that a 1% producer format change dead-letters 1M events.
• Ignoring validation latency — per-record schema checks in hot paths can become the bottleneck.

• For a quarantined record, compare it to the last known-good version of the same type to spot the drift.
• Replay dead-letter topics into the validator to see pass rates improve — that's the 'fix the producer' loop.
• In batch, re-run the failing gate with only the offending rows via a WHERE filter on the violation.
• Use reason-code histograms to find the single most common failure class instead of chasing individual records.