Beginner 12 min read Regression

Regression Metrics

Three numbers, three different stories

MAE reports typical error, RMSE punishes large misses, R2 explains variance-share and can go negative, MAPE adds % readability but breaks near zero — choose by the cost of each kind of error.

MAERMSEMSER2MAPEresidual metricsregression evaluation

Prerequisites: Linear Regression

Before we start

What you will learn

Compute and interpret MAE, RMSE, R2, MAPE
Explain why RMSE > MAE and what the gap means
Choose metrics by business cost of small vs large errors

Introduction

What is this?

Regression metrics summarize residual sizes. MAE = mean |y - ŷ| (typical error, robust to outliers); MSE = mean (y-ŷ)^2 and RMSE = sqrt(MSE) (punish large errors quadratically); R2 = 1 - SS_res/SS_tot (variance explained; can go negative when the model is worse than the mean); MAPE = mean |y-ŷ|/|y| (readable %) but explodes near y=0. Reporting at least RMSE + MAE together tells you whether big errors dominate.

Motivation

Why do you need it?

The metric IS the objective driver: RMSE-ruled models chase the outliers; MAE-ruled models ignore them.
R2 is instantly readable ('the model explains 91% of variance') — and dangerous with extreme outliers.
Different stakeholders need different scales: engineers read RMSE, finance reads MAPE on dollars, ops reads MAE on time.
Choosing blindly produces models tuned for the WRONG error profile.

Think about it

Mental model

Imagine

A judge's scoring rubric: 'any single essay can fail the class'

Think of

RMSE's quadratic penalty

RMSE lets one terrible prediction sink the model's score; MAE averages punishments evenly. Which matches your penalty for a catastrophic miss?

Mechanics

How it works

Compute residuals r_i = y_i - ŷ_i on held-out data only.
MAE = (1/n) sum |r_i| — linear error, outlier-tolerant.
MSE = (1/n) sum r_i^2; RMSE = sqrt(MSE) — emphasizes large |r|.
R2 = 1 - SS_res/SS_tot with SS_tot from the mean predictor; negative R2 = worse than the mean.
MAPE = (1/n) sum |r_i|/|y_i| but undefined/explosive at y=0; prefer WMAPE when zeros are common.

Residual magnitude view.

Residualsy - yhat on holdout

MAEmean absolute

RMSEsqrt of mean square

R21 - SSres/SStot

Hands-on

Examples

The same errors, three readings

import numpy as np
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
 
y = np.array([10, 10, 10, 1000])
yhat = np.array([11, 9, 10, 300])
 
mae = mean_absolute_error(y, yhat)
rmse = mean_squared_error(y, yhat, squared=False)
r2 = r2_score(y, yhat)
 
print("MAE", round(mae, 2))
print("RMSE", round(rmse, 2))
print("R2", round(r2, 3))
print("notes: the single 700 miss dominates RMSE but only 58% of MAE")

MAE understates the big miss's severity while RMSE amplifies it — look at both before trusting either.

Real world

In production

Forecasting: pairs RMSE+MAE; giant periodic misses (promo weeks) drive the gap and signal tail risk.
Latency SLOs: p95/p99 plus RMSE — the tail is what users feel.
Pricing/margin: WMAPE weighted by revenue, so cheap items can't dominate errors.

Guidelines

Best practices

Always report MAE with RMSE; consider MAPE only when the target is strictly positive with a floor; standardize the scale before comparing metrics across models; compute on holdout, never train.

Watch out

Common mistakes

RMSE on heavily skewed targets without log-transform; MAPE with near-zero ground truth producing infinity; R2 as 'accuracy %'; comparing MAE across different units.

When things break

Debugging tips

RMSE/MAE ratio > ~2: investigate residual tail — often a pipeline bug or a heavy-tailed target.
Negative R2: the mean predictor beats you; either the model is broken or the split leaked.
Residual histogram: heavy right tail explains RMSE>MAE.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

RMSE exceeds MAE because:

A target with many true zeros: best metric?

Go deeper

Official documentation

Metric functions (sklearn.metrics.regression)

docs

Coefficient of determination | Wikipedia