MLOps Foundations
Intermediate 15 min read MLOps Foundations

Data & Feature Management

The data side of the system, managed like a product

Data versioning, feature stores and train/serve consistency keep a model's inputs trustworthy: the same feature computed in training and in production, immutable dataset snapshots, and a 'feature as a service' contract.

data versioningfeature storetrain/serve skewfeature pipelinepoint-in-timefeature serving
Prerequisites: The ML Lifecycle Feature Engineering
Before we start

What you will learn

  • Version datasets and features like you version code
  • Explain train/serve skew and how a feature store defeats it
  • Design feature definitions once and serve them everywhere
Introduction

What is this?

Data and feature management is everything that keeps a model's inputs correct and reusable: immutable, versioned dataset snapshots; a feature store where feature definitions live once and are computed the same way for training and serving; point-in-time joins that avoid label leakage; and monitoring of feature distributions. It answers 'what exact data produced this model, and is production consuming the same features?'

Motivation

Why do you need it?

  • Train/serve skew — computing a feature differently at serving than at training — quietly destroys accuracy.
  • Reusing features across teams beats re-implementing them subtly differently in every notebook.
  • Versioned datasets make retraining, backtesting and audits exact and replayable.
  • Feature drift is the earliest warning that a model will degrade.
Think about it

Mental model

Imagine

A company's master product catalogue

Think of

A feature store

One canonical catalogue feeds every storefront and warehouse; nobody pencils in prices from memory. A feature store is the same idea for model inputs: one canonical definition, computed and versioned once, consumed by every model — so retail price (feature) never differs between the shop window (serving) and the stock sheet (training).

Mechanics

How it works

  • Dataset versioning: snapshot raw data with a hash; training runs declare immutable dataset IDs.
  • Feature definitions: one versioned compute unit per feature, shared by training and serving paths.
  • Point-in-time correctness: joins use the value as of the prediction moment, avoiding future 'leakage'.
  • Online vs offline stores: batch-computed offline features for training; low-latency online store for requests.
  • Monitoring: track per-feature distributions and missing rates across train and serve.

One feature definition feeding training and serving.

Feature definitions
Offline store
Online store
Training
Serving
Real world

In production

  • A recommendation team versioned a 2-year click dataset; every model run names the dataset ID, making A/B debug and re-invocations exact.
  • A payments company serves the same fraud features through an online store at <2ms and trains on the offline store — skew dropped to near zero.
  • An anomaly team monitors feature distributions weekly and pages on a drift ratio, catching data pipeline breakage before it reaches models.
Guidelines

Best practices

  • Make every feature a definition (code), not a spreadsheet of values; version the definition.
  • Share the feature compute between training and serving; never hand-roll a second implementation.
  • Use point-in-time joins for any feature that changes over time — prevent leakage by design.
  • Monitor feature stats in training and in production continuously from day one.
  • Keep raw data immutable and reproducible; transforms always consume versioned snapshots.
Watch out

Common mistakes

  • Recomputing a 'feature' differently in the serving request path (missing lag, different rounding).
  • Leaking the future: joining labels or features after the prediction timestamp.
  • Storing feature code only in notebooks — unversioned and divergent.
  • Ignoring missing-rate and distribution drift until models silently degrade.
Root cause analysis

Common issues

Error
Serving scores differ from training-time scores on the same entity
Root cause

Feature computation differs between train and serve (timing, rounding, encoder)

Fix

Use one shared feature definition and log both path outputs on an overlap sample.

Error
Model overfits on validation but not on live traffic
Root cause

Point-in-time violation: features use the future relative to the label

Fix

Rebuild joins strictly 'as of' prediction time; validate on a temporal split.

Error
Feature missing rate spikes in production
Root cause

Upstream pipeline stopped populating the online store for some entities

Fix

Monitor missing/zeroth/NaN rate per feature and alert on the spike; degrade handling.

When things break

Debugging tips

  • Score a sample through the training path and the serving path; any divergence localizes skew.
  • Check the timestamp used in joins — 'as of' bugs cause the classic % of leakages combined.
  • If a model 'suddenly' underperforms, look at feature missing rates and distributions first.
  • Rehydrate an old dataset snapshot to reproduce an incident exactly.
Interviews

Interview checkpoint

Practice

Hands-on lab

Version a Dataset and Detect a Leaky Join
30 min · intermediate
In progress
Objectives
  • Create an immutable dataset snapshot with a content hash
  • Build a 'wrong' join that leaks future information and a 'correct' point-in-time join
  • Prove how leakage inflates offline accuracy

Version a Dataset and Detect a Leaky Join

30 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which symptom most directly indicates train/serve skew?

2

Storing features as manually-maintained spreadsheets is a scalable MLOps practice.

Go deeper

Official documentation

Feature Store best practice (Feast)
docs
Train/serve skew patterns
docs