Model Evaluation
Intermediate 14 min read Model Evaluation

Metrics That Tie to Business

From model scores to decisions worth shipping

Technical metrics decide which model wins; business metrics decide whether the whole thing was worth it. Uplift, expected profit, SLA impact and counterfactual baselines close the loop from model to money.

business metricsupliftexpected valueKPIROIcounterfactual
Prerequisites: Regression Metrics
Before we start

What you will learn

  • Translate model errors into monetary expected value
  • Build an uplift/counterfactual frame for evaluating impact
  • Write an evaluation section a CFO can audit
Introduction

What is this?

Business metrics attach economics to model behavior. Instead of 'F1 = 0.83', they answer 'what did the model earn, save or cost we'd otherwise miss?' Common forms: expected profit per decision, uplift over a baseline (compare against the rule/people the model replaces), opportunity cost of the top false negatives, and SLA/regulatory impact under the promised quality. These metrics are what let a leader say yes to a model.

Motivation

Why do you need it?

  • F1=0.83 tells no one whether to fund the next sprint; $410k saved this quarter does.
  • Every deployment displaces a status quo — the honest question is 'better than what?', not 'is it good?'.
  • Business metrics expose models that score well technically yet add negative value.
  • Model cards with economic metrics survive the quarterly budget review intact.
Think about it

Mental model

Imagine

A race car's lap time vs its value to the team

Think of

Technical vs business metrics

Lap time (accuracy) is why you win the race, but the team is funded on season results — prize money, sponsorships, viewer numbers. Both matter, but they answer different questions.

Mechanics

How it works

  • Define the decision the model feeds and the state of the world without it (baseline/counterfactual).
  • Assign cost/gain weights to outcomes — per TP, FP, FN — from real unit economics.
  • Compute expected value over the population for the model's decision rule and for the baseline rule.
  • Uplift = model value − baseline value; also compute the fraction of gains from the model's incremental decisions.
  • Report alongside a confidence interval or sensitivity analysis over the cost assumptions.

Closing the loop from metrics to money.

Baseline
Model decision
Unit economics
Expected value
Uplift
Real world

In production

  • A collections team compares 'call the model's top 20%' vs 'call everyone' and reports dollars recovered per 1,000 accounts.
  • A recommender reports lift in conversion attributable to the model using an A/B with a rule-based fallback.
  • A CLV-scoring team shows the model's gains concentrated in a segment that previously saw no action at all.
Guidelines

Best practices

  • Define and freeze the baseline before launching the A/B or simulation.
  • Put unit economics in a config so sensitivity analysis is one command.
  • Report error bars: business metrics inherit sampling noise — include a CI.
  • Guard against optimizing the metric the model itself inflates (e.g., conversions influenced by exposure).
Watch out

Common mistakes

  • Reporting 'we saved $100k' computed with model-vs-nothing instead of model-vs-status-quo.
  • Attributing revenue the model merely shifted from another channel.
  • Using point estimates without sensitivity to the cost assumptions.
  • Confusing the technical metric's improvement with the business metric's improvement.
Root cause analysis

Common issues

Error
Model wins on every technical metric but uplift is negative
Root cause

The decision loop it feeds is already near-optimal or costs of action dominate gains

Fix

Recompute with the true production decision rule and action costs before killing the project.

Error
Business owner disputes 'savings'
Root cause

Counterfactual assumptions (what people would have done) are unstated

Fix

Document the baseline rule and run an A/B implemented incrementally to measure real uplift.

When things break

Debugging tips

  • If uplift is suspiciously huge, re-derive the baseline — a too-weak baseline flatters any model.
  • Slice the monetary P&L by segment and by week to find where the model actually moves money.
  • Check that the decision rule in the simulation matches the shipped threshold.
  • Sanity-check a zero-rule (act on nobody) in the simulation to validate the cost model.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Uplift is best defined as:

2

A model must win on technical metrics before business value can be computed.

Go deeper

Official documentation

Google — ML and business value
docs