Bayesian Optimization

• Describe the surrogate + acquisition loop
• Explain exploration-exploitation in sampling
• Use Optuna-style Bayesian search on expensive models

• Expensive models (deep nets, big GBDTs) can't afford thousands of trials — Bayesians make each one count.
• Correlated knobs are exploited naturally (learning rate and depth interact).
• Uncertainty is explicit: you know when you're guessing vs done.
• It typically beats random search 2–5× in trials-to-optimum on smooth surfaces.

You don't grid-search the whole beach. You map readings (surrogate), pick the most promising-but-uncertain spot (acquisition), dig there, update your mental map, and repeat — exploring new beaches only when signal is thin.

• Evaluate a handful of random-start configs to seed surrogate uncertainty.
• Fit the surrogate: score ≈ f(config) with mean and variance predictions.
• Maximize the acquisition function (e.g., expected improvement) over the space to pick the next config.
• Evaluate that config (CV or validation), record score, add to history.
• Refit the surrogate and repeat until the budget or a tolerance bound is hit.

• A deep-learning team tunes lr, batch size, warmup and weight decay with Optuna — ~60 trials instead of grid's ~1,300.
• An AutoML service uses TPE sampling because tree-based surrogates handle mixed categorical/continuous spaces.
• A recurring monthly retrain never needs a full search: the warm-started surrogate re-tunes around the previous optimum.

• Start with the previous best config in the seed set — warm-starting beats cold search.
• Bound the space to plausible regions; wide spaces just slow surrogate refinement.
• Let the surrogate see log-transformed knobs so distances make sense (learning rate magnitudes).
• Add a pruning racing layer (Hyperband-style) when evaluations are trainable iteratively.

• Using Bayesian search on cheap models where random/grid would finish first — overhead dwarfs savings.
• Forgetting to transform log-scale knobs, so the surrogate fits a much harder landscape.
• Over-trusting EI with a tiny surrogate: sample variance collapses too fast and it exploits a false peak.
• Interpreting the surrogate's predicted best as guaranteed — it's predicted, not measured.

• Plot history of observed scores — each trial should mostly land near-or-better than the previous best.
• If acquisition keeps picking boundary values, your ranges are too tight; widen and re-seed.
• Check surrogate predictions vs actuals on a holdout slice of trials to detect surrogate pathology.
• Keep a fixed random-search comparison: if Bayesian isn't beating it, the surface may be noise-dominated — switch budget to folds.