Freshness and Consistency

• Define per-feature freshness SLAs
• Measure online/offline consistency
• Design alerts that catch staleness before it reaches users

• Wrong-freshness values silently degrade model decisions (a stale balance breaks a credit decision).
• Inconsistent live vs training values is training-serving skew hiding in the feature path.
• Explicit SLAs make the platform's staleness a monitored, owned property.
• Consistency checks catch silent bugs in materialization and sync early.

Milk has a sell-by and a cold-chain guarantee from farm to shelf. Every feature has its own sell-by (freshness) and its own cold chain (sync); both broken are detected, and both get pulled from sale (alerted) when violated.

• Each feature declaration records a freshness SLA and a consistency expectation.
• Materialization timestamps values; the online store retains them.
• Freshness checks compare stored-value age to the SLA, alerting on breach.
• Consistency jobs recompute a sample offline and diff against online at the same timestamp.
• Breaches route to the feature owner's alert channel.

• A balance feature has a 15-minute SLA; a sync failure alerts within 10 minutes of breach.
• A nightly consistency job recomputes 1% of online values offline and diffs them.
• A retrain selects feature windows according to the declared SLAs so training matches live freshness.

• Declare freshness per feature at definition time — it becomes the contract.
• Centralize staleness checks in a platform job, not per model.
• Diff online vs offline on a rotating sample to prove consistency.
• Record consistency scores per feature and per day; chase regressions.

• Freshness documented in prose but never checked by automation.
• One global SLA applied to all features regardless of meaning.
• Consistency tests only on green screenshots — no running job, no numbers.
• Ignoring point-in-time in the offline store while building the online server — the two must match for consistent training and serving.

• Staleness alert but 'the pipeline ran' — check the materializer's written-value timestamp, not the job's start time.
• Consistency diff large — compare value semantics, units, and timestamp alignment before suspecting sync.
• Fresh as of corrected by retrain — the SLA and windowing must use the same clock source.