ML Fundamentals
Intermediate 14 min read ML Fundamentals

Model Selection and Tuning

Choose and tune deliberately

Choose a model family by data size, interpretability and non-linearity; tune hyperparameters with grid, random, or Bayesian search — validating each candidate inside the CV loop.

model selectionhyperparameter tuningGridSearchCVRandomizedSearchCVOptunasearch
Prerequisites: Cross-Validation
Before we start

What you will learn

  • Pick a starter model family from dataset size, interpretability and feature scale
  • Run GridSearchCV and RandomizedSearchCV with nested splitting
  • Budget the search and avoid tuning the test set
Introduction

What is this?

Model selection picks a family (linear, tree, ensemble, forest, boosting, NN) from data size, relationship complexity, interpretability needs and latency. Tuning then searches hyperparameters (depth, learning rate, n_estimators, lambda) to maximize validation score. Grid search explores exhaustively, randomized search samples the space, Bayesian (Optuna) learns where the good regions are — all must run inside CV with a fixed budget.

Motivation

Why do you need it?

  • A well-matched family beats a tuned mismatch: linear models crush trees on genuinely linear data with 10k clean rows.
  • Hyperparameters change results as much as the model choice — depth, learning rate and regularization are often the whole game.
  • Search is expensive; a budget and a good prior on ranges save days.
  • Tuning the test set silently; the model card records what was actually searched.
Think about it

Mental model

Imagine

Choosing a car, then tuning the suspension

Think of

Model selection, then hyperparameter tuning

You wouldn't tune a sedan's suspension to off-road specs — pick the vehicle family (selection) first, then tune within that family's valid range.

Mechanics

How it works

  • Selection heuristics: <1k rows → linear/regularized; 1k-100k → boosted trees; interactions/interpretability → trees/GBM; images/text → specialized networks.
  • Define a sensible hyperparameter space from the model's docs (depth 1-12, lr log-uniform).
  • Grid: try every combination — thorough but exponential. Random: sample the space — same expected best for far fewer runs.
  • Bayesian: probe promising regions from past runs (Optuna TPE) — best under budget.
  • Nesting: search on train/valid CV, confirm once on test; share search logs in the model card.

Search strategies.

Grid
Random
Bayesian
Confirm
Hands-on

Examples

RandomSearchCV over a GradientBoosting space

RandomSearchCV over a GradientBoosting space
import numpy as np
from sklearn.datasets import make_regression
from sklearn.model_selection import RandomizedSearchCV, train_test_split
from sklearn.ensemble import GradientBoostingRegressor
from scipy.stats import randint, uniform
 
X, y = make_regression(n_samples=500, n_features=10, noise=10, random_state=0)
 
param_space = {
    "n_estimators": randint(50, 250),
    "max_depth": randint(1, 10),
    "learning_rate": uniform(0.01, 0.3),
    "subsample": uniform(0.6, 0.4),
}
 
search = RandomizedSearchCV(
    GradientBoostingRegressor(random_state=0),
    param_space,
    n_iter=25,
    cv=4,
    scoring="neg_mean_squared_error",
    random_state=1,
    n_jobs=-1,
).fit(X, y)
 
print("best:", search.best_params_)
print("best CV RMSE:", round((-search.best_score_) ** 0.5, 3))

Randomized search samples 25 combos from continuous-ish ranges instead of exploding a grid, and reports the best-validated result without ever touching a test set.

Real world

In production

  • Churn model refresh: a tuned GBM vs hand-picked defaults moved AUC +0.03 while keeping latency under 20ms.
  • Time-boxed sweeps: 50 Optuna trials nightly over stale features keeps deployment drift-compensated.
  • Fraud: a tuned threshold inside the search loop (threshold as a hyperparameter) beats tuning the model only.
Guidelines

Best practices

  • Search inside CV and confirm on a frozen holdout; put a hard time/trial budget in the config; log all trials, not just the winner; inspect the search landscape for flat 'good-enough' plateaus before committing.
Watch out

Common mistakes

  • Tuning on test (leak); grid over absurd ranges that waste budget; no nested validation so the reported score is optimistic; too many trials relative to data so search starts tuning noise.
When things break

Debugging tips

  • Plot trial scores vs each hyperparameter — flat lines mean the range was uninformative.
  • Compare 'best' vs 'defaults' — if they're equal, the extra tuning budget was wasted.
  • Check search verbosity/logs for convergence or repeated best on the edge of the range (range too small).
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

When is RandomizedSearchCV preferable to exhaustive grid?

2

Tuning on the test set leaks because:

Go deeper

Official documentation

GridSearchCV API
docs
RandomizedSearchCV API
docs
Optuna: hyperparameter optimization framework
docs