Concept Drift
Intermediate 15 min read Concept Drift

Model Refreshing Strategies

When and how often should the model catch up?

Model freshness is a strategy, not a reflex: schedule-driven, drift-triggered, or continuous updating each trades staleness-latency against cost and stability. The right policy matches drift velocity, label availability, and business tolerance.

retraining policycontinuous trainingscheduled retraindrift-triggeredfreshness
Prerequisites: Concept Drift Overview MLOps Lifecycle
Before we start

What you will learn

  • Compare schedule-driven, drift-triggered, and continuous refreshing
  • Match the strategy to drift velocity and label economics
  • Gate refreshes behind evaluation so the update is also an improvement
Introduction

What is this?

Model refreshing is a deliberate cadence policy — schedule-driven (daily/weekly), drift-triggered (alerts gate a rerun), or continuous/streaming (updates with each example) — balanced against cost, label lag, evaluation gating, and stability regression risk.

Motivation

Why do you need it?

  • Stale models quietly degrade; fresh models cost money and churn risk.
  • The right freshness depends on drift velocity and label economics.
  • Updating teaches stability debt — a fresh model can surprise you.
  • Gate updates with evaluation so freshness never overrides quality.
Think about it

Mental model

Imagine

Refreshing your map app on a train vs on foot

Think of

Retraining cadence

On a train the world (drift) moves fast — refresh (continuous/streaming updates) matter every stop. On foot, daily pulls (scheduled) suffice, and pulling every second (retraining on noise) just burns battery for worse maps.

Mechanics

How it works

  • Measure or estimate drift velocity and label latency.
  • Pick the cadence: scheduled, drift-triggered, or streaming.
  • Decide what the update window includes (retrain, fine-tune, or warm-start).
  • Gate the candidate with offline+online evaluation and a canary.
  • Inspect for high drift in the window so the refresh matches current reality.

Choosing and running a refresh strategy.

Drift velocity
Labels/ICE cost
Cadence
Train on window
Evaluate + canary
Real world

In production

  • A fraud model refreshing hourly with batch windows; fraud concept moves quickly.
  • A credit model on scheduled monthly refresh; drift is slow and labels lag 60 days.
  • An inventory forecast refresh-triggered by a drift alert rather than a fixed clock.
Guidelines

Best practices

  • Match cadence to drift velocity and label economics, not FOMO.
  • Warm-start from the serving model to avoid throwing away what still holds.
  • Always evaluate the candidate against the drift-window reality before shipping.
  • Ship via canary with a rollback; measure online before global rollout.
Watch out

Common mistakes

  • Retraining on a fixed clock into stale labels that institutionalize the old concept.
  • Continuous updating that chases noise and churns the model every hour.
  • Replacing a stable model with a 'fresh' one that regressed offline but 'felt right'.
  • Full cold retrains that waste compute and lose the serving model's knowledge.
When things break

Debugging tips

  • Fresh model underperforms the old — the training window captured old concept or label lag; check the window's drift.
  • Refresh never improves — drift is slow; switch to a longer cadence and watch.
  • Refresh crashes online — evaluation didn't match the live window; gate with a canary first.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Warm-starting a refresh means…

2

A fast retraining cadence is always better for accuracy.

Go deeper

Official documentation

MLflow — retraining & windows
docs