Download as PDF, PPTX
![Intro Ian Dewancker. SigOpt for ML: TensorFlow ConvNets on a Budget with Bayesian Optimization. Ian Dewancker. SigOpt for ML: Unsupervised Learning with Even Less Supervision Using Bayesian Optimization. Ian Dewancker. SigOpt for ML : Bayesian Optimization for Collaborative Filtering with MLlib. #1 Trusting the Defaults Keras recurrent layers documentation #2 Using the Wrong Metric Ron Kohavi et al. Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained. Xavier Amatriain. 10 Lessons Learning from building ML systems [Video at 19:03]. Image from PhD Comics. See also: SigOpt in Depth: Intro to Multicriteria Optimization. #4 Too Few Hyperparameters Image from TensorFlow Playground. Ian Dewancker. SigOpt for ML: Unsupervised Learning with Even Less Supervision Using Bayesian Optimization. #5 Hand Tuning On algorithms beating experts: Scott Clark, Ian Dewancker, and Sathish Nagappan. Deep Neural Network Optimization with SigOpt and Nervana Cloud. #6 Grid Search NoGridSearch.com References - by Section](https://image.slidesharecdn.com/commonproblemsinhyperparameteroptimization21-170324153628/75/Common-Problems-in-Hyperparameter-Optimization-30-2048.jpg)
Uploaded bySigOpt
Common Problems in Hyperparameter Optimization
This document discusses common problems that can occur in hyperparameter optimization including trusting default values too much, using the wrong evaluation metric, overfitting hyperparameters to validation data, optimizing too few hyperparameters, relying too heavily on manual tuning, using grid search which scales poorly, using random search which has high variance, and provides recommendations such as using Bayesian optimization which can efficiently search large hyperparameter spaces and intelligently determine the most promising configurations to evaluate next.
In this document
Powered by AI
Introduction to hyperparameters and their tuning for model selection, aiming for optimal values.
Showcases significant accuracy improvements across different machine learning models with specified percentages.
Highlights the problem of depending on default hyperparameter values which may not suit all models.
Emphasizes the importance of selecting appropriate metrics to balance goals and avoid biases.
Discusses the risks of overfitting and methods for metric generalization like cross-validation.
Importance of not neglecting hyperparameters and feature parameters during optimization for better results.
Critiques manual tuning and advocates for using algorithms for optimization to save time and cost.
Descriptions of grid search limitations through examples of model evaluations across hyperparameters.
Examines random search effectiveness, its variances, and the lack of intelligence in the method.
Introduces advanced methods for optimization, mentioning genetic algorithms and Bayesian optimization.
Overview of using SigOpt's Bayesian optimization service for defining and reporting hyperparameters.
Wraps up the presentation with acknowledgments and references to further reading related to the discussed topics.
Common Problems in Hyperparameter Optimization
- 1. Common Problems inHyperparameter Optimization Alexandra Johnson @alexandraj777
- 2.
- 3. Hyperparameter Optimization ● Hyperparameter tuning,model tuning, model selection ● Finding "the best" values for the hyperparameters of your model
- 4. Better Performance ● +315%accuracy boost for TensorFlow ● +49% accuracy boost for xgboost ● -41% error reduction for recommender system
- 5. #1 Trusting theDefaults
- 6. ● Default valuesare an implicit choice ● Defaults not always appropriate for your model ● You may build a classifier that looks like this: Default Values
- 7. #2 Using theWrong Metric
- 8. Choosing a Metric ●Balance long-term and short-term goals ● Question underlying assumptions ● Example from Microsoft
- 9. Choose Multiple Metrics ● ●Composite Metric ● Multi-metric
- 10.
- 11. Metric Generalization ● Crossvalidation ● Backtesting ● Regularization terms
- 12. Metric Generalization ● Crossvalidation ● Backtesting ● Regularization terms
- 13. Metric Generalization ● Crossvalidation ● Backtesting ● Regularization terms
- 14. #4 Too FewHyperparameters
- 15. Optimize all Parametersat Once
- 16.
- 17.
- 18. Example: xgboost ● Optimizedmodel always performed better with tuned feature parameters ● No matter which optimization method
- 19.
- 20. What is anOptimization Method?
- 21. You are notan Optimization Method ● Hand tuning is time consuming and expensive ● Algorithms can quickly and cheaply beat expert tuning
- 22. Grid Search RandomSearch Bayesian Optimization Use an Algorithm
- 23.
- 24. No Grid Search Hyper- parameters Model Evaluations 2100 3 1,000 4 10,000 5 100,000
- 25.
- 26. Random Search ● Theoreticallymore effective than grid search ● Large variance in results ● No intelligence
- 27. Use an IntelligentMethod Genetic algorithms Bayesian optimization Particle-based methods Convex optimizers Simulated annealing To name a few...
- 28. SigOpt: Bayesian OptimizationService Three API calls: 1. Define hyperparameters 2. Receive suggested hyperparameters 3. Report observed performance
- 29.
- 30. Intro Ian Dewancker. SigOptfor ML: TensorFlow ConvNets on a Budget with Bayesian Optimization. Ian Dewancker. SigOpt for ML: Unsupervised Learning with Even Less Supervision Using Bayesian Optimization. Ian Dewancker. SigOpt for ML : Bayesian Optimization for Collaborative Filtering with MLlib. #1 Trusting the Defaults Keras recurrent layers documentation #2 Using the Wrong Metric Ron Kohavi et al. Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained. Xavier Amatriain. 10 Lessons Learning from building ML systems [Video at 19:03]. Image from PhD Comics. See also: SigOpt in Depth: Intro to Multicriteria Optimization. #4 Too Few Hyperparameters Image from TensorFlow Playground. Ian Dewancker. SigOpt for ML: Unsupervised Learning with Even Less Supervision Using Bayesian Optimization. #5 Hand Tuning On algorithms beating experts: Scott Clark, Ian Dewancker, and Sathish Nagappan. Deep Neural Network Optimization with SigOpt and Nervana Cloud. #6 Grid Search NoGridSearch.com References - by Section
- 31. References - bySection #7 Random Search James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Ian Dewancker, Michael McCourt, Scott Clark, Patrick Hayes, Alexandra Johnson, George Ke. A Stratified Analysis of Bayesian Optimization Methods. Learn More blog.sigopt.com sigopt.com/research