What is the best strategy for Hyperparameter Tuning in 2024?

All answers to this question.

GridSearch is definitely outdated for complex models like XGBoost because it evaluates every single combination, many of which are useless. RandomSearch is statistically proven to find better results in less time because it explores the search space more broadly. However, for a professional workflow in 2024, Bayesian Optimization is the gold standard. It uses the results of previous iterations to "guess" where the next best parameters might be. Tools like Optuna are fantastic because they allow for "Pruning"—automatically stopping trials that show poor performance early on, which saves a massive amount of compute.

   Answered 2024-02-15 by Jennifer Smith

Since you are using XGBoost, have you considered using its built-in "cv" function with early stopping first to narrow down the number of boosting rounds before tuning other parameters?

   Answered 2024-02-20 by Michael Brown

• Michael, I haven't combined the "cv" function with my tuning script yet. That makes a lot of sense—it would let me fix the n_estimators based on the learning rate first. This would significantly reduce the dimensions of my search space. I was trying to tune everything at once, which is probably why it was so slow. I’ll try fixing the boosting rounds first and then use Optuna for the max_depth and subsample parameters.

     Commented 2024-02-24 by Robert Taylor

Don't forget that sometimes the default parameters are surprisingly good. Only tune the parameters that actually impact the "Bias-Variance" tradeoff significantly for your specific data.

   Answered 2024-03-02 by Sarah Wilson

• Totally agree with Sarah. Over-tuning can sometimes lead to a model that is too specific to your training set and performs poorly in production.

     Commented 2024-03-05 by Jennifer Smith