The process of adjusting the hyperparameters to improve a model’s overall fit, performance, accuracy, and other measures. Approaches to hyperparameter tuning include:
- Grid Search: Iteratively trying a bunch of pre-defined hyperparameters for the best performing hyperparameter. Brute-force search.
- Random Search: Doing grid search, but randomly and not as exhaustively. Faster and can outperform grid search when there are only a small number of hyperparameters.
- Bayesian Optimization: A probabilistic model that maps hyperparameter values to the target evaluated on a validation set.
- Gradient-Based Optimization: Gradient is calculated using hyperparameters and then optimized using gradient descent.
- Evolutionary Optimization: Advanced techniques that can yield more optimal hyperparameters.
