ML Foundations
Intermediate 12 min read ML Foundations

Baselines & Benchmarks

A simple model is the first truth-teller

Before any fancy model, measure simple baselines — mean prediction, majority class, trivial heuristics. A baseline that beats your model means the problem framing is wrong, not the architecture.

baselinebenchmarkmajority classheuristicsanity check
Prerequisites: Regression & Classification
Before we start

What you will learn

  • Define simple baselines for regression and classification
  • Use baselines to validate your framing and data
  • Judge whether a complex model actually earns its keep
Introduction

What is this?

A baseline is the simplest decent predictor: the mean for regression, the majority class for classification, a keyword rule for search, copy-paste for generation. Baselines validate that your data, features and task framing are sound, and they quantify what a 'real' model must beat to be worth its complexity. Teams that skip baselines celebrate models that can't beat 'always guess Friday'.

Motivation

Why do you need it?

  • A model that can't beat the mean isn't learning anything the data supports.
  • Baselines reveal framing bugs early: if a heuristic wins, you may have built the wrong task.
  • They set a floor for what complexity must justify — cost, latency, maintenance.
  • For LLM apps, a 'retrieve nearest clause' baseline is the reference a RAG system beats.
Think about it

Mental model

Imagine

Measuring against the room average

Think of

Using a baseline model

You judge a candidate against the room-average performance, not your hopes. Anyone below the average is just noise; you only hire (deploy) those clearly above it.

Mechanics

How it works

  • Regression: predict the training mean (or median). Metric = the score that simple prediction achieves.
  • Classification: predict the most common class. Per-class metrics expose imbalance.
  • Heuristic baselines: lexicons, keyword matching, recency rules — cheap, interpretable floors.
  • LLM baselines: no retrieval (model-only), top-1 chunk, copy-source-chunk — set context for RAG gains.
  • The real model must beat the baseline materially, with statistical significance.

Baseline-first workflow.

Data
Baseline
Measure
Real model
Ship decision
Real world

In production

  • A churn model must beat 'assume no churn' by +10 precision points before anyone adopts it.
  • A RAG prototype compares answer quality against 'echo the top retrieved chunk verbatim'.
  • A search re-ranker gates itself against 'order by recency' on the same offline queries.
Guidelines

Best practices

  • Build the baseline before the model; it's also your first E2E-test harness.
  • Measure the baseline on the exact eval set and metric the real model will face.
  • Share baseline numbers in every experiment report; they set context for gains.
  • Re-measure baselines when data changes — baselines drift with distribution.
Watch out

Common mistakes

  • Skipping baselines and reporting '80% accuracy' that a majority baseline also hit.
  • Comparing a model to an outdated baseline or a different metric.
  • Celebrating a 0.5% gain over baseline that noise explains away.
Root cause analysis

Common issues

Error
Model equals majority-class baseline
Root cause

Imbalanced data or weak features for the minority class

Fix

Fix imbalance/label coverage; report per-class, not aggregate.

Error
Baseline accuracy rose between weeks
Root cause

Distribution shift makes the majority even more dominant

Fix

Track class priors; a rising trivial baseline is a drift alarm.

Error
RAG gains vanish vs 'echo chunk' baseline
Root cause

Retrieval returns irrelevant top chunks

Fix

Debug retrieval precision before blaming generation.

When things break

Debugging tips

  • Model ≈ baseline: check for a data problem (label noise, leakage) before model choice.
  • Baseline in production different from eval — the distribution moved, so re-baseline.
  • If a heuristic beats everything, the feature signal needed may not be in the data at all.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

You're evaluating a classifier on 90/10 imbalanced data. Which baseline applies?

2

A model that barely beats the baseline justifies its complexity on its own.

Go deeper

Official documentation

sklearn dummy estimators
docs