Random Search

• Randomize over distributions, not lists
• Explain why random search wins for low effective dimension
• Grow budgets deliberately, not to the biggest grid

• Grids spend most cells repeating similar values across an irrelevant knob.
• With a budget of N fits and D knobs, grid gets ~N^(1/D) values per knob; random search independently samples all N values per knob.
• Real tuning problems are low-dimensional in effect even when the space is wide.
• It's parallel, reproducible (seeded), and gives clean uncertainty-free reasoning about budget use.

Walking every 10-unit cell of a 100×100 floor uses enormous effort to cover each corridor once; darts thrown at random eventually land everywhere, and with equal darts you've scattered across every region rather than marching a lattice.

• Define a distribution per knob: log-uniform for magnitudes, uniform/choice for discrete, integers for counts.
• Sample N configs jointly from the product distribution (independent draws).
• Evaluate each config with CV; keep the best mean score.
• As budget grows, the empirical best approaches the true opt within the space.
• Optionally refine around the best sample with a second tighter round.

• A gradient-boosting pipeline random-searches 60 configs (lr, depth, subsample, colsample) once, cheaply, instead of 6^4 = 1296 grid cells.
• A neural-net trainer randomizes lr and weight-decay with a log-uniform sampler and 40 trials per tuning cycle.
• Tuning-as-a-service systems sample the same pseudo-random seed for reproducibility audits.

• Use log-uniform for scale knobs (learning rate, strength) — equal chance per decade, not per linear step.
• Fix the sampler seed and log the exact samples with the experiment.
• Spend the budget in two waves: coarse global, then 2–3x denser around the promising region.
• Add a no-op 'dummy knob' check: if it gets ranked top often, the score is noise-dominated.

• Sampling uniform over [0, 1] for multiplicative knobs — a 0.001 learning rate gets sampled as often as 0.999.
• Reusing the grid's value lists and sampling from them — you inherit the grid's poor spacing.
• Randomly sampling seeds/batch sizes where sensitivity is fake, wasting budget.
• Believing more samples always find better; at some point you're resampling the same plateau.

• Plot score vs each knob from the logged samples to see which knobs actually move the metric.
• If the top-10 configs scattered across unconnected regions, the response surface is noisy — increase folds, not samples.
• Compare a 30-sample random run to a 30-cell grid run on the same budget; the winner is usually clear.
• Ensure all samples are logged (config + fold scores) — the distribution of scores teaches more than the argmax.