Parameters vs Hyperparameters

• Distinguish learned parameters from preset hyperparameters
• Map common hyperparameters to the behavior they control
• Avoid treating learnable settings as hyperparameters (and vice versa)

• Wrong hyperparameters can make a good architecture perform terribly, and no amount of training fixes it.
• Tuning is how you find the trade-off between underfitting and overfitting for your specific data.
• Knowing the search space bounds the work — not everything is tunable.
• Model comparison means nothing unless hyperparameters were tuned at comparable effort.

The recipe quantities and oven temperature you set before cooking are hyperparameters; the chemical state of the finished dish is what the process produced — parameters. You can tweak the recipe, but you can't decide the dish's texture by fiat.

• Before training, define the model family and its hyperparameter values (depth, lr, penalty).
• Training runs an optimizer over parameters given fixed hyperparameters — gradient descent moves the weights.
• After training, the parameters are known; the hyperparameters remain exactly as you set them.
• Choose hyperparameters by evaluating on validation (CV), since training loss falls with them in ways that overfit quickly.
• Too-large capacities (deep trees, big nets, wide feature spaces) overfit; too-small underfit — the search finds the middle.

• A GBDT pipeline tunes max_depth, learning_rate and colsample in CI; batch size and seeds stay fixed.
• A deep-learning service fixes architecture and tunes only lr, weight decay and dropout — a bounded, debuggable space.
• Regression teams tune penalty (ridge/lasso strength) and the solver, leaving the fitted coefficients to training.

• Treat hyperparameter search as part of the model build, and validate every hyperparameter choice.
• Group tunables: data-level, model-level, training-level — tune in stages to keep the space book length sane.
• Tie the search budget to the compute you can afford; a wider space with a 10-trial search is worse than a focused one with 50.
• Log hyperparameters with every experiment — a model without its config is meaningless.

• Tuning the random seed and calling it a hyperparameter; it should be fixed per experiment.
• Believing hyperparameter 'defaults' are optimal for your dataset — they rarely are.
• Over-tuning on the test set (selecting settings until test score peaks), then shipping a model with no honest estimate.
• Confusing an optimizable hyperparameter with a learnable parameter the data already informs.

• If validation is flat across settings, the model family is the problem, not the tuning.
• A hyperparameter whose effect flips between runs is probably insufficiently regularized or data too small.
• Print parameter count vs hyperparameter settings to see capacity changes you intended.
• Run the 'default config' capture — sometimes defaults are a fine baseline and tuning adds noise.