Freshness and Timeliness SLAs

• Define freshness as a lag metric with an explicit reference clock
• Monitor freshness continuously rather than at batch boundaries
• Escalate to the right owner when a table misses its freshness SLA

• Stale data is often invisible at the row level — no error, just age. Only a clock can detect it.
• Freshness is the SLA most likely to cause business harm: a stale risk report is worse than no report.
• Defining per-table freshness kills the myth that 'the pipeline is fine' when only some tables update.
• Continuously sampled lag catches stalls mid-window instead of at the end of the batch.

The store doesn't check milk only when the truck arrives — every shelf is scanned continuously and expired stock is pulled. Data freshness monitoring scans each table's 'expiration' on a cadence, not just at batch end.

• Choose a reference clock: event time (when the fact occurred) or run time (when the pipeline ran).
• Define lag = now - newest event_time in table (or schedule - last successful run).
• Agree the threshold per table per use case: ml feature (minutes), warehouse dim (hours), regulatory (calendar T+1).
• Run a heartbeat monitor on a fixed cadence that samples lag and emits an alert on breach.
• Add output columns: last_success, current lag, breach history — so consumers self-serve freshness.
• Escalate by ownership: paging the table's producer, not the platform team, when freshness slips.

• A real-time fraud team alerts when kafka→warehouse lag exceeds 5 minutes, catching not just pipeline failures but dead consumers.
• A retail warehouse marks a table 'STALE' in the catalog when its heartbeat hasn't ticked for two scheduled windows.
• A finance org encodes T+1 as a freshness check: if the day's data isn't present by 05:30 local, the batch is marked failed and on-call is woken.

• Always use event time for freshness when consumers reason about facts, not about pipeline runtime.
• Store a last_update heartbeat per table and monitor that heartbeat's age — no heartbeat, no trust.
• Set the sampling cadence tighter than the SLA (e.g. sample every 5 min for a 1 h SLA).
• Expose freshness in the data catalog so consumers see the number before they query.

• Measuring freshness only at batch completion — a stall that starts at 02:00 isn't seen until 06:00.
• Using run time to mark a table 'fresh' when its newest event is days old (empty input looks healthy).
• Hard-failing a dashboard query because one optional table is stale — freshness should page, not block reads.
• One global freshness policy — a 5-minute table and a monthly report get the same (wrong) treatment.

• Separate ingestion lag (source→landing) from transform lag (landing→published): each has a different fix.
• Check the heartbeat's last value and timestamp to see whether the monitor itself is alive.
• When lag jumps, look at the run history for the table: did a DAG step run but write nothing?
• Cross-check event-time max with the source's monotonic id to confirm the clock isn't drifting.