Production ML
Advanced 14 min read Production ML

Learning from Live Feedback

Your model edits its own training data

Models change the world they predict: recommenders shape what people click, fraud models shape what fraudsters do. Feedback loops can be virtuous or vicious, and label leakage hides it. Closing the loop deliberately is a design problem, not an afterthought.

feedback looplabel leakageselective labelsself-fulfillingover-fitting to trafficexploration
Prerequisites: Monitoring Models Retraining Strategies
Before we start

What you will learn

  • Recognize the ways a model's decisions alter its own future labels
  • Identify selective-label and exploration gaps that bias retraining
  • Design deliberate loop closures: logging, unbiased labels, exploration, shadow
Introduction

What is this?

A feedback loop means the model's outputs affect the data used to train the next model. A recommender that shows users only what it predicts they'll click shapes future click labels; a fraud model that blocks some transactions preserves the label for what it blocks but not for what it allowed; an autocomplete changes what people type next. Left undetected, these loops make the model over-fit to its own taste, hide its misses and create self-confirming regressions that offline look like gains.

Motivation

Why do you need it?

  • Reward hacking: recommending 'safe' content maximizes clicks today but collapses exploration and long-term relevance.
  • Selective labels: a fraud model only confirms disputes on what it flagged — its misses are invisible by construction.
  • Self-fulfilling prophecy: a churn model that triggers retention offers 'cures' churn the model then can't learn to predict.
  • The loop compounds: small biases at deployment become large, baked-in biases at the third retrain.
Think about it

Mental model

Imagine

An echo chamber in a room of mirrors

Think of

A model with an unmanaged feedback loop

You ask the room 'what do I like?' and every mirror reflects the last answer you gave it. If the mirrors (the model's outputs feeding back as inputs/labels) are never disturbed, the answer converges on itself — loudly, confidently, and wrong about the outside world.

Mechanics

How it works

  • Log decisions + outcomes with the input snapshot, and record detector states (model version, thresholds that fired).
  • Build unbiased label paths: randomized or exploration traffic, holdout slices, or delayed ground truth from sources independent of the decision (reviews, chargebacks, surveys).
  • Detect loop pathologies: check for drops in outcome variance, rising model confidence at constant base rates, and degradation of exploration coverage.
  • Design the agent: add exploration in action space (epsilon-greedy, Thompson sampling) for recommender/decision models.
  • Retrain on loops-aware data: sample over time, weight decision-dependence, and never train purely on 'what the previous model showed'.

The three ways feedback reaches back into training.

Model decides
World changes
Labels appear
Loop pathology
Next retrain
Real world

In production

  • A recommender keeps 5% of traffic on exploration ranks so click labels stay informative about the long tail and drift is measurable.
  • A fraud team logs all decisions and uses chargebacks (independent of the model's flag) rather than only disputed-flagged cases, debiasing its label set.
  • An autocomplete team A/B-tests interventions and measures the pre-intervention text distribution, isolating the model's influence on the feedback it receives.
Guidelines

Best practices

  • Add permanent exploration to every decision model whose outputs become labels.
  • Use label sources independent of the decision (outcomes, audits) to avoid selective-label censoring.
  • Log AND snapshot inputs with every decision — loop analysis needs the full context, not just scores.
  • Track outcome variance and model confidence together; a confidence spike at flat base rates screams self-confirmation.
  • Periodically audit 'would we have sampled this the same way with the older model?' to surface conditioning.
Watch out

Common mistakes

  • Retraining on clicks produced by the previous model's recommendations as if they were independent evidence.
  • Measuring fraud model quality only on the transactions it flagged — miss rates are censored.
  • Infinite-confidence spirals: no exploration, so the model 'learns' to echo and the retrain solidifies it.
  • Deleting decisions after serving to 'save storage', destroying the ability to audit the loop.
Root cause analysis

Common issues

Error
Model confidently recommends a shrinking set of items
Root cause

No exploration; feedback loop collapses the action space

Fix

Add exploration (epsilon-greedy / Thompson) and monitor coverage of the item space.

Error
Fraud model precision high, recall unknowable
Root cause

Selective labels: only flagged transactions get disputes

Fix

Pull independent outcome sources (chargebacks, audits) and sample unflagged flows for review.

Error
Retrained model worse than the frozen version
Root cause

Retrain data reflects the old model's decisions (loop bias), no exploration labels

Fix

Mix time-weighted data, add unbiased/exploration slices, and backtest against a pre-loop holdout.

When things break

Debugging tips

  • Rebuild the label cohort by decision: did the model choose the outcome, or did the world?
  • Check outcome variance across ranked deciles — a flat tail indicates the model stopped exploring.
  • Compare retraining-window metrics vs a static-baseline window to see loop contributions.
  • Use shadow/was-already-planned interventions to estimate counterfactual outcomes when feasible.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Selective labels occur when:

2

A recommender needs permanent exploration to keep its own training data honest.

Go deeper

Official documentation

Closed-loop learning & bandits (NIPS tutorial)
docs
Google — Rules of ML (feedback loops)
docs