Business Metric Tracking
From model truth to business value
Model metrics are means; business metrics are the ends. Track downstream impact — revenue, retention, cost per outcome, SLA compliance — and bind model quality to business value so stakeholders and engineers share a single source of truth.
What you will learn
- Map model quality to downstream business outcomes
- Build a north-star metric tree that ties ML to value
- Reconcile model and business dashboards
What is this?
Business metric tracking connects model quality to what leadership optimizes — revenue, retention, cost per good outcome, SLA compliance — through a metric tree where each ML input maps to a measurable business leaf, reviewed by model and business owners together.
Why do you need it?
- A good AUC with bad economics is still bad — value is the north star.
- Business stakeholders need one shared truth, not model jargon.
- Metric trees expose which ML component moves the top line.
- Reconciliation catches 'nice model, worse business' mistake early.
Mental model
FC/DC: a chef's taste vs the restaurant's revenue
Model metric vs business metric
Food judges score taste (model metric — high). If revenue falls despite great taste, review where value breaks: price, portions, marketing funnel. Both the taste panel and the P&L are watched; the P&L is what keeps the lights on.
How it works
- Define the north star: revenue, margin, retention, cost-per-good-outcome.
- Build the tree: model outputs → decisions → customer outcomes → business leaf.
- Instrument each edge so attribution drills from leaf to model input.
- Schedule cadence with model + business owners together — shared verdicts.
- Bind release gates to business impact, not just AUC deltas.
North-star metric tree.
In production
- A Credit model's gating shows default-rate improvement but customer-acquisition loss — reconciling the trade-off in front of both teams.
- A Recommender links ranking changes to GMV and engagement, not just CTR deltas.
- An Ops team tracks cost-per-resolved-ticket as the business leaf of its triage model.
Best practices
- North star explicit; every model maps to a leaf.
- Model metric and business metric reviewed in the same meeting, by the same people.
- Release gates cite business impact estimates, not only AUROC.
- Reconcile attribution: get the 'which line moved' drill-down right.
Common mistakes
- Reporting AUC to a P&L owner or revenue to the data team — language mismatch.
- Model and business dashboards disagree because nothing reconciles them.
- Chasing a model-metric lift that the business leaf never felt.
- Attribution guesswork — refreshing dashboards without an edge-mapping drill.
Debugging tips
- Model improved, business flat — the improvement didn't survive the decision layer; check conversion/decision step.
- Business metric moved by a non-model rollout — isolate with the metric tree's edge mapping.
- Leaders ask 'so what' — map your release to a concrete leaf change, not an abstract lift.
Interview checkpoint
Quick quiz
The north star of a metric tree should be…
A higher model metric always implies a better business number.