In this talk, I describe some newest Hyperparameter optimization libraries in Python. Discuss the pros and cons of them and help you choose the one for you.

1. Hyperparameter Optimization
(landscape) in Python
kuba@neptune.ml
@NeptuneML
https://medium.com/neptune-ml
Jakub Czakon

2. ● Slides link on Twitter @NeptuneML
● Blog posts on Medium @jakub.czakon:
○ Part 0 intro
○ Part 1 Scikit-Optimize
● Experiments on Neptune experiments link
Materials

3. learning rate
depth
feature fraction
Model scoredata
https://bbabenko.github.io/convolutional-learnings/

4. learning rate
depth
feature fraction
Model score
bin_nr
groupby columns
lagging
Feature
Engineering
imputation method
scaling method
Data
Cleaning
thresholds
Post-
processing
data
objective(params, data=data) -> score

5. ● Grid search
● Random search
● Guided search (Bayesian Optimization)
● Grad student search (still best)
Methods

6. ● Scikit-Optimize
● Optuna
● HpBandSter
● Hyperopt (sort of)
● … many more
Libraries

7. ● API / ease of use
● Documentation
● Speed / Parallelization
● Visualization suite
● Experimental results
Evaluation Criteria

8. Let’s Start

9. Scikit-Optimize

10. API
search space
+
objective
+
{fun}_minimize

11. API: search space
● Basic options:
○ skopt.space.Real
○ skopt.space.Integer
○ skopt.space.Categorical
● No support for nested search spaces

12. API: search space
SPACE = [skopt.space.Real(0.01, 0.5, name='learning_rate', prior='log-uniform'),
skopt.space.Integer(1, 30, name='max_depth'),
skopt.space.Integer(2, 100, name='num_leaves'),
skopt.space.Integer(10, 1000, name='min_data_in_leaf'),
skopt.space.Real(0.1, 1.0, name='feature_fraction', prior='uniform'),
skopt.space.Real(0.1, 1.0, name='subsample', prior='uniform'),
]

13. API: objective
● Define a function to minimize!
● Decorate if you want to keep parameter names

14. API: objective
def objective(**params):
return -1.0 * train_evaluate(X, y, **params)
@skopt.utils.use_named_args(SPACE)
def objective(**params):
return -1.0 * train_evaluate(X, y, **params)

15. API: {fun}_minimize
● A few optimizers to choose from
○ skopt.forest_minimize
○ skopt.gbrt_minimize
○ skopt.gp_minimize
● Accepts callbacks

16. API: {fun}_minimize
results = skopt.forest_minimize(objective, SPACE,
callback=[monitor],
n_calls=100,
n_random_starts=10,
base_estimator='ET',
acq_func='LCB',
xi=0.02,
kappa=1.96)

17. API: {fun}_minimize
def monitor(res):
neptune.send_metric('run_score', res.func_vals[-1])

18. API: {fun}_minimizeAPI: {fun}_minimize
objective(params) -> score
surrogate(params) -> est_score
surrogate(params2) =
est_score*2
surrogate(params1000) =
est_score1000
objective(params2) = score2
surrogate(params1) =
est_score*1
expensive
cheap
explore cheap
try expensive

19. API: {fun}_minimize
● There are (hyper)hyperparameters
● Acquisition function:
○ ‘EI’, ‘PI’ , minimal improvement or probability of improvement
○ ‘LCB’, what is the max expected improvement
● Exploration vs exploitation
○ xi for ‘EI’ and ‘PI’, low xi exploration high xi exploitation
○ Kappa for ‘LCB’, low kappa exploitation, high kappa exploration

20. Documentation
● Amazing!
● Functions have docstrings.
● A lot of examples.
link

21. Visualizations
● Options:
○ skopt.plots.plot_convergence - score improvement
○ skopt.plots.plot_evaluations - space search evolution
○ skopt.plots.plot_objective - sensitivity
● Beautiful and very useful.

22. Visualizations: plot_convergence
skopt.plots.plot_convergence(results)

23. Visualizations: plot_convergence
skopt.plots.plot_convergence(results_list)

24. Visualizations: plot_evaluations
skopt.plots.plot_evaluations(results)

25. Visualizations: plot_evaluations
skopt.plots.plot_evaluations(results)

26. Visualizations: plot_evaluations
skopt.plots.plot_evaluations(results)
random search forest_minimize

27. Visualizations: plot_evaluations
skopt.plots.plot_objective(results)

28. Speed & Parallelization
● Runs sequentially and you cannot distribute it across many machines
● You can parallelize base estimator at every run with n_jobs
● If you have just 1 machine it is fast

29. Experimental Results
● Example training script is available here.
● I grid searched over acquisition functions, xi and kappa.
● All the experiments are available in Neptune with a tag skopt

30. Experimental Results

31. Conclusions: good
● Easy to use API and great documentation
● A lot of optimizers and tweaking options
● Awesome visualizations
● Solid gains over the random search
● Fast if you are running sequentially on 1 machine

32. Conclusions: bad
● Search space doesn’t support nesting
● No support for distributed computing

33. Optuna

34. API
search space & objective
+
{fun}_minimize

35. API: search space & objective
def objective(trial):
params = OrderedDict([
('learning_rate',trial.suggest_loguniform('learning_rate', 0.01, 0.5)),
('max_depth',trial.suggest_int('max_depth', 1, 30)),
('num_leaves',trial.suggest_int('num_leaves', 2, 100)),
('min_data_in_leaf',trial.suggest_int('min_data_in_leaf', 10, 1000)),
('feature_fraction',trial.suggest_uniform('feature_fraction', 0.1, 1.0)),
('subsample',trial.suggest_uniform('subsample', 0.1, 1.0))])
score = -1.0 * train_evaluate(X, y, params)
return score

36. API: search space & objective
● Basic options:
○ suggest_categorical
○ suggest_int , suggest_discrete_uniform
○ suggest_uniform , suggest_loguniform
● Nested search spaces with ifs

37. API: search space & objective
def objective(trial):
classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest'])
if classifier_name == 'SVC':
svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10)
classifier_obj = sklearn.svm.SVC(C=svc_c)
else:
rf_max_depth = int(trial.suggest_loguniform('rf_max_depth', 2, 32))
classifier_obj = sklearn.ensemble.RandomForestClassifier(max_depth=rf_max_depth)
…

38. API: {fun}_minimize
● Selects based on Tree Parzen Estimation (like hyperopt) paper
● Allows pruning
● Handles exceptions in objective
● Doesn’t have callbacks

39. study = optuna.create_study()
study.optimize(objective, n_trials=100])
results = study.trails_dataframe()
API: {fun}_minimize

40. API: {fun}_minimize: pruning
from optuna.integration import LightGBMPruningCallback
def objective(trial):
params = OrderedDict([
('max_depth',trial.suggest_int('max_depth', 1, 30)),
('num_leaves',trial.suggest_int('num_leaves', 2, 100))])
pruning_callback = LightGBMPruningCallback(trial, 'auc')
score = -1.0 * train_evaluate_with_pruning(X, y, params, pruning_callback)
return score
def train_evaluate_with_pruning(X, y, params, callback):
...
model = lgb.train(params, train_data, ... , callbacks = [pruning_callback])
return model.best_score['valid']['auc']

41. Documentation
● Solid read-the-docs project,
● Docstrings, docstrings everywhere,
● A lot of examples.
link

42. Visualizations

43. Speed & Parallelization
● Can be easily distributed across one or many machines
● Has pruning to speed up unpromising runs

44. Speed & Parallelization: one
study.optimize(objective, n_trials=100, n_jobs=5)

45. Speed & Parallelization: many
…
study = optuna.Study(study_name='distributed-search', storage='sqlite:///example.db')
study.optimize(objective, n_trials=100)
...
$ optuna create-study --study-name "distributed-search" --storage "sqlite:///example.db"
$ python optuna_search.py
$ python optuna_search.py
terminal 1
terminal 2
terminal 3
optuna_search.py

46. Speed & Parallelization: many (aws)
$ optuna create-study --study-name "distributed-search" --storage "DB_ADDRESS"
$ python optuna_search.py
$ python optuna_search.py
instance 1
instance 2
instance 3
$ python optuna_search.pyinstance 1
create ECS cluster
create RDS DB instance
launch ECS instances
setup

47. Experimental Results
● Example training script is available here.
● No (hyper)hyperparameters to search
● All the experiments are available in Neptune with a tag optuna

48. Experimental Results

49. Conclusions: good
● Easy to use API
● Great documentation
● Can be easily distributed over a cluster of machines
● Has pruning
● Search space supports nesting

50. Conclusions: bad
● Only TPE optimizer available
● No visualizations
● *No gains over the random search (with 100 iterations budget)

51. Optuna is hyperopt with:
● better api
● waaaay better documentation
● pruning
● exception handling
● simpler parallelization

52. HpBandSter

53. HpBandSter

54. HpBandSter
● Budget options:
○ Dataset size
○ Number of epochs
○ Time
○ Number of features

55. ● HyperBand on Steroids
● It has state-of-the-art algorithms
○ Hyperband paper
○ BOHB (Bayesian Optimization + Hyperband) paper
● Distributed-computing-first API
HpBandSter

56. API
server
+
worker
+
optimizer

57. API
server
+
worker
+
optimizer

58. API: server
● Workers communicate with server to:
○ get next parameter configuration
○ send results
● You have to define it even for the most basic problems (weird)

59. API: server
import hpbandster.core.nameserver as hpns
NS = hpns.NameServer(run_id=RUN_ID, host=HOST, port=PORT, working_directory=WORKING_DIRECTORY)
ns_host, ns_port = NS.start()

60. API: worker: objective
from hpbandster.core.worker import Worker
class TrainEvalWorker(Worker):
...
def compute(self, config, budget, working_directory, *args, **kwargs):
loss = -1.0 * train_evaluate(self.X, self.y, budget, params)
return ({'loss': loss,
'info': { 'auxiliary_stuff': 'worked'
}
})

61. API: worker: search space
● Basic options:
○ CSH.{Categorical/Ordinal}Hyperparameter
○ CSH.{Uniform/Normal}IntegerHyperparameter
○ CSH.{Uniform/Normal}FloatHyperparameter
● Nested search spaces with ifs

62. API: worker: search space
class TrainEvalWorker(Worker):
...
@staticmethod
def get_configspace():
cs = CS.ConfigurationSpace()
learning_rate = CSH.UniformFloatHyperparameter('learning_rate',
lower=0.01, upper=0.5, default_value=0.01, log=True)
subsample = CSH.UniformFloatHyperparameter('subsample',
lower=0.1, upper=1.0, default_value=0.5, log=False)
cs.add_hyperparameters([learning_rate, subsample])
return cs

63. API: worker: connecting to server
worker = TrainEvalWorker(run_id=RUN_ID, nameserver=ns_host, nameserver_port=ns_port)
worker.run(background=True)

64. API: optimizer
from hpbandster.optimizers import BOHB
optim = BOHB(configspace = worker.get_configspace(),
run_id = RUN_ID,
nameserver=ns_host,
nameserver_port=ns_port,
result_logger=NeptuneLogger(),
eta=3, min_budget=0.1, max_budget=1,
num_samples=64, top_n_percent=15,
min_bandwidth=1e-3, bandwidth_factor=3)
study = optim.run(n_iterations=100)

65. API: optimizer: callbacks
class NeptuneLogger:
def new_config(self, *args, **kwargs):
pass
def __call__(self, job):
neptune.send_metric('run_score', job.result['loss'])
neptune.send_text('run_parameters', str(job.kwargs['config']))

66. Documentation
● Decent Read-the-docs project,
● Missing docstrings in a lot of places,
● A bunch of examples.
link

67. Visualizations
● Options:
○ hpvis.losses_over_time - score improvement
○ hpvis.concurrent_runs_over_time - speed/parallelization
○ hpvis.finished_runs_over_time - budget adjustment
○ hpvis.correlation_across_budgets - budget adjustment
○ hpvis.performance_histogram_model_vs_random - sanity check
● Very lib/debug-specific but can be useful for tweaking

68. Visualizations:
losses_over_time
all_runs = results.get_all_runs()
hpvis.losses_over_time(all_runs);

69. Visualizations:
correlation_across_budgets
hpvis.correlation_across_budgets(results);

70. Visualizations:
performance_histogram_model_vs_random
all_runs = results.get_all_runs()
id2conf = results.get_id2config_mapping()
hpvis.performance_histogram_model_vs_random(all_runs, id2conf);

71. Speed & Parallelization
● Can be easily distributed across threads/processes/machines

72. Speed & Parallelization: threads
workers=[]
for i in range(N_WORKERS):
w = TrainEvalWorker(run_id=RUN_ID, id=isleep_interval = 0.5,
nameserver=ns_host, nameserver_port=ns_port)
w.run(background=True)
workers.append(w)
optim = BOHB(configspace = TrainEvalWorker.get_configspace(),
run_id = RUN_ID,
nameserver=ns_host,
nameserver_port=ns_port)
study = optim.run(n_iterations=100, min_n_workers=N_WORKERS)

73. Speed & Parallelization: processes
workers=[]
for i in range(N_WORKERS):
w = TrainEvalWorker(run_id=RUN_ID, id=isleep_interval = 0.5,
nameserver=ns_host, nameserver_port=ns_port)
w.run(background=False)
exit(0)
optim = BOHB(configspace = TrainEvalWorker.get_configspace(),
run_id = RUN_ID,
nameserver=ns_host,
nameserver_port=ns_port)
study = optim.run(n_iterations=100, min_n_workers=N_WORKERS)

74. Speed & Parallelization: machines
Follow the example from the docs … but it is not obvious

75. Experimental Results
● Example training script is available here.
● A lot of (hyper)hyperparameters to search
● All the experiments are available in Neptune with a tag hpbandster

76. Experimental Results

77. Conclusions: good
● State-of-the-art algorithm
● Can be distributed over a cluster of machines
● Visualizations
● Search space (kind of) supports nesting

78. Conclusions: bad
● Complicated API
● Missing docstrings

80. Results (mostly subjective)
Scikit-Optimize Optuna HpBandSter Hyperopt
API/ease of use Great Great Difficult Good
Documentation Great Great Ok(ish) Bad
Speed/Parallelization Fast if
sequential/None
Great Good Ok
Visualizations Amazing None Very lib specific Some
*Experimental results 0.8566 (100) 0.8419 (100)
0.8597 (10000)
0.8629 (100) 0.8420 (100)

81. Dream library
Viz + Callbacks Scikit-Optimize
+
API + Docs + Pruning + Parallelization Optuna
+
Optimizers HpBandSter

82. Conversions between
results objects are in
neptune-contrib
import neptunecontrib.hpo.utils as hpo_utils
results = hpo_utils.optuna2skopt(study)
Dream library

83. ● If you don’t have a lot of resources - use Scikit-Optimize
● If you want to get SOTA and don’t care about API/Docs - use HpBandSter
● If you want good docs/api/parallelization - use Optuna
Recommendations

84. Data science work sharing hub.
Track | Organize | Collaborate
jakub@neptune.ml
@NeptuneML
https://medium.com/neptune-ml
Jakub Czakon