Early Stopping and Pruning

• Apply early stopping with a validation patience
• Compare pruning mid-run candidates vs finishing all
• Use Hyperband-style racing to spend budget on the best

• Full training runs dominate tuning budgets; most never win.
• Validation curves stabilize early — the ranking of candidates barely changes mid-run.
• Early stopping is also a regularizer: halting before overfit gives better validation performance.
• Hyperband can solve tuning 5–10x faster for iterative learners (GBDT, NN).

You don't give every candidate a full year on probation. Early promising work buys more runway; clear mismatches are cut in week one. The strongest finalist still gets the complete process — some runners just never go the full distance.

• Early stopping: monitor validation loss each epoch; if no improvement in P epochs (patience), roll back to the best checkpoint and stop.
• Restore-the-best is important — stopping at a low point is the point.
• Pruning: search candidates report scores at fixed milestones; a candidate ≥ a distance below the incumbent best is terminated.
• Hyperband: try R candidates with 1 unit of budget each, keep the top 1/η, double budget, repeat bands — batting averages dominate finishing.
• Report best config + the compute saved (a KPI for tuning efficiency).

• An XGBoost trainer early-stops on `validation_evaluations` with early_stopping_rounds=50, restoring the best iteration.
• A torch training loop checks `best_val < val - eps` each epoch and saves the best weights, halting at patience.
• Optuna's MedianPruner + HyperbandSampler cut an 8-hour search to 90 minutes on a deep-learning job.

• Always restore the best epoch/checkpoint, not the last weight vector.
• Use a validation split/curve that doesn't leak into the search's decisions (a racing budget is still model selection).
• Pick patience proportional to loss noise: noisier losses need bigger patience to avoid premature stops.
• Combine pruning with Bayesian sampling — the sampler learns which regions deserve the surviving budget.

• Early stopping without restoring the best state — dropping to the tail of the curve.
• Pruning on noisy per-epoch metric where one lucky batch kills a good config.
• Using the tuned-validation data for early stopping AND for the final report — it gets reused, so it's a guide, not gospel.
• Hyperband with too-frequent pruning, so no candidate ever reaches full budget and all underfit.

• Plot validation curves (metric vs epoch) — if the 'best epoch' is the last, patience was too short.
• When pruning kills everything, lower the pruning threshold or reduce frequency of checks.
• Log stopped-epoch + reason for every candidate; audits love that trace.
• Reproduce the best config with a full-unpruned run once to confirm the pruned winner wasn't budget-starved.