Time Series
Advanced 15 min read Time Series

Evaluating Forecasts

Is this forecast better than last-value, and how do we know?

Forecast errors must be measured against the right baseline and the right scale. RMSE, MAE, MASE, coverage and per-horizon profiles tell you whether a forecast is actually good — MASE and coverage especially when naive baselines are hard to beat.

MASEforecast evaluationbacktestcoverageprediction intervalnaive baselinesMAPE
Prerequisites: Cross Validation for Time Series Regression Metrics
Before we start

What you will learn

  • Choose RMSE/MAE vs scale-free MASE/sMAPE
  • Evaluate distributional honesty with interval coverage
  • Backtest coarsely and per-horizon
Introduction

What is this?

Forecast evaluation compares predictions against observed future values from a backtest, with errors normalized to answer 'better than what?'. Core quantities: MAE/RMSE (scale-bound), MASE (MAE divided by the naive one-step MAE — <1 means better than persistence), sMAPE (symmetric % error), and interval coverage (did the 90% band contain 90% of realized values). Good forecasting teams report point-error AND interval honesty together — a great point forecast with lying intervals is a broken tool.

Motivation

Why do you need it?

  • Raw RMSE hides whether you beat 'forecast last value' — often the real baseline.
  • Scale-free metrics let you compare across SKUs, geos and seasons.
  • Intervals drive safety-stock and budget sizing; point-only evaluation under-builds buffers.
  • Per-horizon profiles expose where the model's edge ends.
Think about it

Mental model

Imagine

A weather service judged on 'was the 20% rain call right 20% of the time?'

Think of

Interval coverage plus point error

Saying 'it will rain' is only useful if you ALSO get the probability right. Forecasts are the same: the point line and its uncertainty band are judged together, and the band being honest is half the job.

Mechanics

How it works

  • Run a walk-forward backtest: retrain/refresh at origin o, forecast h ahead, record error, roll on.
  • Compute MAE/RMSE on the stacked errors; compute MASE by dividing by the in-sample naive MAE.
  • Check coverage: fraction of true values inside each claimed interval band (e.g., 90% band ≈ 90%).
  • Slice errors by horizon (h=1, h=7, h=30) and by calendar season to find regime dependence.
  • Declare improvement only if the winner beats baselines on scale-free metrics with margin beyond fold noise.

Walk-forward backtest feeds several error lenses.

Origins
Forecasts
Point errors
Interval check
Verdict
Real world

In production

  • An inventory team gates a forecast deploy on MASE < 1.0 on the last 26 weeks and 90% coverage in [87%, 93%].
  • A finance forecaster reports sMAPE quarterly, with per-line and per-horizon tabled breaks.
  • An ML forecast pipeline compares LightGBM vs SARIMA on walk-forward RMSE AND MASE — two lenses for the same trade.
Guidelines

Best practices

  • Always report against a naive baseline (seasonal naive preferred for cyclic data).
  • Backtest with 3+ origins and a horizon span covering the operational look-ahead.
  • Report point and interval metrics; coverage should sit inside its own CI.
  • Keep the evaluation window's calendar continuity (skip holiday noise or flag it).
Watch out

Common mistakes

  • Choosing the last week as the 'test' and treating luck as victory.
  • Using sMAPE when values cross zero — the symmetric absolute error blows up near 0.
  • Measuring point error only, then trusting the fitted bands for safety stock.
  • Comparing models on different backtest windows or origins — apples to oranges.
Root cause analysis

Common issues

Error
Point MAE great, coverage terrible
Root cause

Interval variance estimated from the wrong scale (stationary assumption)

Fix

Re-fit interval widths on backtest residuals per horizon; widen bands proportionally to scale.

Error
RMSE says model B wins, MASE says A wins
Root cause

B has fewer big errors (RMSE-ish) but worse typical error vs naive

Fix

Report both and decide by the business cost of big misses.

When things break

Debugging tips

  • Zoom to a 2×2 grid: (model good/bad) × (coverage good/bad) — four different stories needing four fixes.
  • If MASE ≈ 1, the model is a fancy persistence; don't ship it as an upgrade.
  • Coverage too high (99% on a 90% band)? Intervals too fat — likely variance overestimated.
  • Slice by horizon: h=30 skills dying → your features lack the slow memory or calendar terms.
Interviews

Interview checkpoint

Practice

Hands-on lab

Backtest: MASE, Coverage, and Per-Horizon Errors
30 min · advanced
In progress
Objectives
  • Run a rolling-origin backtest for two models
  • Compute MASE against the seasonal-naive baseline
  • Check interval coverage for a forecast band

Backtest: MASE, Coverage, and Per-Horizon Errors

30 min · advanced

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

MASE below 1.0 means:

2

A single hold-out week is a sufficient forecast backtest.

Go deeper

Official documentation

MASE — Hyndman's forecasting textbook
docs