Model Performance Monitoring
Intermediate 16 min read Model Performance Monitoring

Online Evaluation and Proxies

Real predictions meet real outcomes — mess and all

Online evaluation trades offline purity for reality: real predictions face real outcomes, scored on matured labels or outcome proxies when ground truth lags. It's the honest operating metric, full of bias and delay you design around.

online metricslabel waitingproxy outcomelive evaluationmaturity
Prerequisites: Offline Evaluation Revisited Concept Drift
Before we start

What you will learn

  • Score models on live outcomes with label-waiting windows
  • Use outcome proxies with explicit time-to-signal
  • Fairly compare live performance across periods
Introduction

What is this?

Online evaluation measures live-model quality: predictions matched to matured outcomes (or outcome proxies with known time-to-signal) and scored window-by-window, including delay-adjusted comparisons so 'now' is never judged on immature labels.

Motivation

Why do you need it?

  • Online reveals what reality actually delivered.
  • Proxies compress label lag from weeks to minutes.
  • Maturation windows stop fake incidents from immature readers.
  • Comparisons become fair and operational, not aspirational.
Think about it

Mental model

Imagine

Restaurant reviews: instant ratings vs the Michelin review

Think of

Proxies vs matured labels

Google maps ratings (proxies) appear instantly and reflect feel; the Michelin review (label) takes months but is ground truth. Both count — you route the instant proxy to catch breakdowns and respect the matured label for decisions.

Mechanics

How it works

  • Predict → store → wait the maturity window for labels.
  • Choose proxies where labels lag: CTR for recommenders, refunds for fraud-adjacent, engagement for chat.
  • Score both proxy-now and label-later metrics in one dashboard.
  • Maturity-calibrate comparisons: same window age, same segment.
  • Keep the proxy-to-label gap visible — divergence means proxy drift.

Online evaluation with maturation.

Prediction
Proxy signal
Maturation
Labels mature
Score + reconcile
Real world

In production

  • A Recommender scores CTR instantly and conversion at +7 days — both live.
  • A Fraud model tracks provisional-flag rates live and confirmed-fraud rates after review.
  • A Credit model watches early-default lanes now and matured book performance quarterly.
Guidelines

Best practices

  • Instrument predictions with IDs so labels can always be joined back.
  • Publish proxy-now and label-later side by side with their windows.
  • Reconcile proxy-to-label drift quarterly; a moved proxy is a moving target.
  • Freeze the maturity window — publication discipline beats optimistic metrics.
Watch out

Common mistakes

  • Judging live 'now' on immature labels — manufactured incidents.
  • Choosing a proxy whose drift mimics model quality noise (ops-induced).
  • Comparing today's numbers to yesterday's on different maturity ages.
  • Ignoring the proxy/label gap, so dashboards lie progressively.
When things break

Debugging tips

  • Metric dips then climbs back — likely label maturation, not the model; look at window age.
  • Proxy up, label down — the proxy itself moved (feature rollout?); decorrelate and re-baseline it.
  • Online numbers swing every release — comparison cohort differs; align segments and maturity.
Interviews

Interview checkpoint

Practice

Hands-on lab

Score the Live Model
40 min · intermediate
In progress
Objectives
  • Store predictions and join labels after a maturity window
  • Compute proxy and label-later metrics
  • Diagnose a fake incident from immature labels

Score the Live Model

40 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The maturation window solves…

2

A good proxy should also be monitored for its own drift.

Go deeper

Official documentation

Weights & Biases — model evaluation
docs