Training Pipelines
Intermediate 16 min read Training Pipelines

Hyperparameter Sweeps

Search with a budget, measure with integrity

Sweeps are orchestrated hyperparameter search: grid, random, Bayesian or bandit. Done right they need budgets, early stopping and honest evaluation — done wrong they're just gaming the tracker.

hyperparameter tuningsweepbayesian optimizationearly stoppingsearch budget
Prerequisites: Pipeline Orchestration Tracking Metrics and Parameters
Before we start

What you will learn

  • Choose a search strategy by compute budget and dimension count
  • Configure early stopping to save compute safely
  • Evaluate sweep results on a never-touched test set
Introduction

What is this?

A hyperparameter sweep runs many training jobs over a configuration search space — grid, random or adaptive (Bayesian/bandit). Its purpose is to maximize a target metric within a compute budget, while keeping evaluation honest by selecting on validation, not the final test set.

Motivation

Why do you need it?

  • Hand-tuning burns time and takes exactly one path; sweeps explore the space systematically.
  • Adaptive methods (Bayesian, bandit) explore where they can learn, saving compute.
  • Early stopping cuts failed configurations early with little risk.
  • Tracked sweeps produce comparable, auditable configuration → metric maps.
Think about it

Mental model

Imagine

Fishing with nets of different meshes

Think of

Search strategies

A grid is a fixed mesh (every point); random casts wide with cheap probes; Bayesian reads where the previous catches were and re-casts nearby. Each has a catch-per-effort — budget decides which to fish with.

Mechanics

How it works

  • The space is declared (ranges or discrete for each hyperparameter).
  • Sampler selects configurations: grid (all), random (fixed budget), Bayesian (model-informed).
  • Each configuration trains on a budget slice with tracked params and metrics.
  • Early stopping prunes unpromising runs (median stopping, patience).
  • Results rank on a fixed validation set; the winner is evaluated once on the test set.

A sweep loop with early stopping and honest final eval.

Define space
Sample
Train each
Early stop
Rank on validation
One test eval
Hands-on

Examples

A cheap random sweep with tracking

A cheap random sweep with tracking
for trial in range(20):
    cfg = {"lr": 10 ** random.uniform(-4, -1),
           "layers": random.choice([2, 3, 4])}
    with mlflow.start_run():
        mlflow.log_params(cfg)
        model, val = train_and_val(**cfg)
        mlflow.log_metric("val_loss", val)
# then: choose min val_loss, evaluate ONCE on test

Random search over log-scaled lr and discrete layer counts, tracked per trial, with the final test evaluation reserved for the winner.

Real world

In production

  • A transformer team budgets 200 GPU-hours; Bayesian search halved the configs needed.
  • A tabular pipeline integrates a sweep stage that early-stops median-poor runs.
  • A registry promotion records the sweep's config map as evidence for the champion's choice.
Guidelines

Best practices

  • Set a compute budget before the sweep: trials × cost per trial.
  • Prefer random over grid when some dimensions are low-value.
  • Use early stopping where costs are high; keep it safe by a minimum training floor.
  • Select on validation; touch the test set only once.
Watch out

Common mistakes

  • Selecting the winner on test — the test set stops being honest.
  • Grid over a dimension that barely matters, exploding cost exponentially.
  • Early stopping with too low a floor, killing eventual-winner configs.
  • Forgetting to log seeds, so sweep results look noisy for the wrong reasons.
When things break

Debugging tips

  • 'Sweep found great val but champ is bad' — check whether the win was on test vs val.
  • Trials crash at random — same seed, different config? Look for config-induced edge cases.
  • Sweep cost estimate blown — add per-trial budget accounting before launch.
Interviews

Interview checkpoint

Practice

Hands-on lab

Run a Budgeted Random Sweep
30 min · intermediate
In progress
Objectives
  • Define a hyperparameter space and budget
  • Run a tracked random sweep with early stopping
  • Select a winner on validation and evaluate once

Run a Budgeted Random Sweep

30 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Over many trials, early stopping saves compute mainly by:

2

Picking the best config from the test set is an acceptable selection strategy.

Go deeper

Official documentation

Optuna docs
docs
scikit-learn — model selection
docs