Model Evaluation
Intermediate 14 min read Model Evaluation

Regression Metrics

Every error is a number; how you average it is a philosophy

MAE, MSE, RMSE, R² and MAPE each weight errors differently. Choosing the right one depends on whether big misses are worse than many small ones and on the scale of what you are predicting.

MAEMSERMSER2regression metricsMAPE
Prerequisites: Metrics Fundamentals
Before we start

What you will learn

  • Compute and interpret MAE, MSE, RMSE and R²
  • Choose an error metric to match the business cost of error magnitude
  • Diagnose distributions where a single regression metric lies
Introduction

What is this?

Regression metrics summarize the errors between continuous predictions and targets. MAE (mean absolute error) averages absolute differences; MSE (mean squared error) averages squared differences; RMSE is its square root; R² measures how much variance the model explains relative to a mean baseline; MAPE expresses error as a percentage of the target. Because they weight big errors differently, they are not interchangeable.

Motivation

Why do you need it?

  • A single 1,000-unit blunder changes RMSE dramatically but barely moves MAE — which behavior do you want to pay for?
  • R² is the standard 'did it beat predicting the mean' summary for stakeholder reports.
  • MAPE lets you compare models across differently-scaled targets, but misbehaves near zero.
  • The metric you minimize at training should match the metric you report at review.
Think about it

Mental model

Imagine

Speed cameras vs. average-speed checkpoints

Think of

MAE vs RMSE

One harshest judge is the worst single stretch of a journey; the other spreads blame across the whole trip. MSE-like metrics fire heavily on any huge spike; MAE-like metrics care about the total.

Mechanics

How it works

  • For each prediction y_hat and truth y, the per-row error is |y_hat-y| (MAE) or (y_hat-y)² (MSE).
  • MAE = mean absolute error; RMSE = sqrt(mean squared error); both are in the target's units.
  • Because squares dominate, RMSE ≥ MAE always; the gap measures how much the tail errors matter.
  • R² = 1 - SS_res/SS_tot, where SS_res is residual variance and SS_tot is the variance around the mean; 1 is perfect, 0 matches the mean predictor, negative is worse than the mean.
  • MAPE = mean(|y_hat-y|/|y|) × 100; it is scale-free but explodes when y is near zero.

Error metric choice through the lens of one big outlier.

Errors
MAE
RMSE
MAPE
R²
Hands-on

Examples

Show how one outlier moves each metric

Show how one outlier moves each metric
python3 -c "
y = [10, 12, 9, 14, 11]
pred = [10, 12, 9, 14, 999]   # one bad prediction
import numpy as np
y = np.array(y); pred = np.array(pred)
mae = np.abs(pred - y).mean()
rmse = np.sqrt(((pred - y)**2).mean())
print('MAE', round(mae, 2), 'RMSE', round(rmse, 2))
# remove the outlier
pred2 = pred.copy(); pred2[-1] = 11
print('without outlier  MAE', round(np.abs(pred2-y).mean(), 2), 'RMSE', round(np.sqrt(((pred2-y)**2).mean()), 2))
"

One bad point inflates RMSE to 394×MAE. Which model looks 'fine' depends entirely on which metric the reviewer glances at.

Real world

In production

  • A pricing team reports MAPE to the business because 'we're within 4%' is understandable — but engineers also track RMSE for tail pricing disasters.
  • A forecast system optimized with RMSE because missing a demand spike by 300 units is far costlier than missing many days by 5.
  • A churn-score regression uses MAE to tolerate moderate noise and avoid chasing outliers that are data-entry garbage.
Guidelines

Best practices

  • Report MAE and RMSE together — the gap reveals tail sensitivity.
  • Check R² against a sensible baseline (mean, last value, historical median), not zero.
  • Use percentage error (MAPE) only when targets are comfortably away from zero.
  • Inspect residuals vs fitted values; a nice RMSE can sit on a biased, funnel-shaped error cloud you'd miss.
Watch out

Common mistakes

  • Drawing conclusions from RMSE alone when the error distribution is skew — the median error may be far smaller.
  • Using MAPE with near-zero targets: true value 0.5, prediction 1.0 is '100% error'.
  • Saying 'R² = 0.9' without noting it is computed on a transformed or winsorized target.
  • Comparing RMSE across problems with different target scales as if they were comparable.
Root cause analysis

Common issues

Error
R² is negative
Root cause

Model fits worse than predicting the mean (or constant predictions on skewed data)

Fix

Treat negative R² as 'worse than mean baseline'; investigate target transform or a constant-value model.

Error
MAPE shows 400% on one row
Root cause

A target near zero makes the percentage explode

Fix

Use WMAPE (weighted by |y|) or clip targets; MAPE is not defined for zero targets.

When things break

Debugging tips

  • Plot residuals vs predictions: a wedge shape means heteroskedasticity, not a metric bug.
  • If RMSE >> MAE, hunt for outliers — decide whether they're real signal or measurement noise.
  • Compute metrics per segment (by region, by day) alongside the global one.
  • Benchmark your model's MAE against the trivial 'predict the median' on the same fold.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which metric punishes a single huge error the most?

2

An R² of 0.95 means the model explains 95% of the variance in the target.

Go deeper

Official documentation

sklearn regression metrics
docs