Real-Time Governance and Skew
Fast answers still need an audit trail
Real-time serving inherits governance duties: version registration, auditability of predictions, and guarding against feature skew between training and live. Skew detection and governance hooks close the loop that fast serving otherwise skips.
What you will learn
- Log predictions with enough metadata to audit later
- Detect and quantify train-serve feature skew
- Wire real-time serving into model governance
What is this?
Real-time governance ensures every served prediction is attributable — the model version, inputs, decision and time are logged — and that live features match what training expected. Skew detection scrapes that gap. Governance is what turns fast serving from an engine into a managed system.
Why do you need it?
- An un-logged fast answer is a liability no one can reconstruct later.
- Feature skew silently degrades live quality while offline metrics look fine.
- Regulated domains need per-decision audit records.
- Governance hooks (version policy, blacklists, approvals) apply in real-time too.
Mental model
A controller logging every ATC decision, not just the flights
Prediction governance
Fast decisions are useless for review if nothing's recorded. The controller logs the flight, decision, conditions and version of the screen (model) — so a post-incident review can replay exactly what was seen, not what was assumed.
How it works
- Serving middleware stamps every prediction with model version, request hash, features snapshot, time.
- A sample (or full) prediction log lands in a governed store.
- Skew checks compare live feature distributions to the training baseline.
- Alerts/providers feed drift metrics to dashboards and retrigger gates.
- Governance policy (versions allowed, approvals required) is enforced in the serving config.
Governance wrapping real-time serving.
In production
- A credit model logs every decision with the feature snapshot for regulator review.
- A skew monitor flags that live users now send a feature whose train range never covered.
- Serving rejects requests for a model version blacklisted after a recall.
Best practices
- Sample full predictions; log every decision for high-risk domains.
- Snapshot the features seen, not a reference to volatile state.
- Run skew checks on live vs baseline distributions continuously, not monthly.
- Align train and serve feature definitions to one source of truth.
Common mistakes
- Logging predictions without the feature snapshot — unreconstructable later.
- Skew checks on aggregate columns while row-level drift hides inside bins.
- Governance policy changed in config but never read at request time.
- Ignoring new-feature risk: live canaries quietly push features outside training range.
Debugging tips
- Served values look off vs train — print live-train feature diffs per field; skew often hinges on one feature.
- Audit gap — prediction logs missing request hash mean you can't tie decisions back to inputs.
- Policy not enforced — verify governance config made it into the running artifact, not just CI.
Interview checkpoint
Hands-on lab
- Snapshot live features per prediction
- Compare live vs training distributions per field
- Emit alerts when skew breaches thresholds
Build a Skew Watchdog
35 min · advanced
Quick quiz
The most common root of 'offline fine, live broken' is…
High-risk domains should log every decision, even at the cost of volume.