Sagemaker Automatic model tuning

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Soji Adeshina, Machine Learning Engineer, Amazon AI
SageMaker Automatic Model
Tuning

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Roadmap
• Hyperparameters
• Search Based HPO
• Bayesian HPO
• Amazon SageMaker AMT

Hyperparameters

What is a Hyperparameter
• Hyperparameter = algorithm parameter
• Training algorithm accepts hyperparameter(s) and returns model
parameters
• It affects how an algorithm behaves during model training process
• “Any decision an algorithm author can’t make for you”

Examples of Hyperparameters
Model:
Number of layers: 1, 2, 3, …
Activation functions: Sigmoid, tanh, RELU, …
Optimization:
Method: SGD, Adam, AdaGrad, …
Learning Rate: 0.01 to 2
Data:
Batch Size: 8, 16, 32 …
Augmentation: Resize, Normalize, Color Jitter, …

Model vs Hyperparameter Optimization
𝑙∗
= min
𝜃
ℎ(𝜃)
ℎ(𝜃) = min
𝑤
𝑓(𝑤|𝑋, 𝑦, 𝜃)
Optimize Model params (𝑤)
Optimize Hyperparams (𝜃)

Blackbox Optimization
• We aim to minimize the objective function .
• We have no knowledge of what the objective function is.
• We don’t have access to the gradients of the objective function.
• All we know is what goes into the function and what comes out.
ℎ( 𝜃)

Search Based HPO

Grid Search
Learning Rate
Activation
Sigmoid
RELU
tanh
0 20.5 1 1.5

Grid Search - Shortcomings
• In grid search the user specifies a finite set of values for each hyperparameter.
• Each hyperparam increases degree of freedom and results in combinatorial explosion.
• Assume each hyper-param has 5 options
e.g. Learning Rate: 0, 0.5, 1, 1.5, 2
1 HP = 5 combinations
2 HPs = 5*5 = 25 combinations
3 HPs = 5*5*5 = 125 combinations
…
10 HPs = 5^10 = 9,765,625 combinations
N HPs = 5^N combinations

Grid Search - Shortcomings
Learning Rate
Activation
Sigmoid
RELU
tanh
0 20.5 1 1.5
Some hyper-params more important than others.

Grid Search
Learning Rate
Activation
Sigmoid
RELU
tanh
0 20.5 1 1.5
Wasted Compute

Random Grid Search
Learning Rate
Activation
Sigmoid
RELU
tanh
0 20.5 1 1.5

Bayesian HPO

Model based Bayesian HPO
Learning Rate
Activation
RELU
0 20.5 1 1.5
ℎ 𝜃 : 𝑡𝑟𝑢𝑒 (ℎ𝑖𝑑𝑑𝑒𝑛)
𝐷: 𝑠𝑎𝑚𝑝𝑙𝑒𝑠
ℎ′ 𝜃 : 𝑎𝑝𝑝𝑟𝑜𝑥𝑖𝑚𝑎𝑡𝑒
𝑐: 𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒
• ℎ 𝜃 is expensive so use an approximation or surrogate model ℎ′(𝜃) instead
• Use an acquisition function 𝔼[𝐼 𝜆 ] to selects next points

• Keeps track of previous evaluations and infers expected behaviour.
• It is Bayesian in a sense that the surrogate model model uses prior probability
distribution to make predictions about the posterior.
𝑃 𝑌 𝑋 ∝ 𝑃 𝑌 𝑋 𝑃(𝑌)
• Improves our beliefs about the objective function by applying iterative learning.
Model based Bayesian HPO

Surrogate Model - Gaussian Process
• Gaussian Process is a distribution over functions each of which returns mean and variance of a
Gaussian distribution.
𝑓: 𝒳 → ℝ
𝑓(𝑋𝑡1
), 𝑓(𝑋𝑡2
), … , 𝑓(𝑋𝑡 𝑛
)~𝒩(𝝁, 𝜮)
• Gaussian distribution is a distribution of random numbers that is described by mean 𝜇 and variance
𝜎2
.
• Each distribution corresponds to a set of hyperparameters Λ;
𝜆𝑖 𝜖Λ = 𝑖=1
𝑛
Λ 𝑖
• A Gaussian process is fully specified by a mean 𝜇 𝜆 and a covariance function 𝑘(𝜆, 𝜆′).
𝒢(𝜇 𝜆 , 𝑘(𝜆, 𝜆′))

Gaussian Process for model of model loss

Covariance Matrix
Similarity between 2 points: controls ‘smoothness’.
SageMaker uses Matérn kernel with 𝜐 = 5/2

Acquisition Function
• Given posterior distribution of functions…
𝔼 𝕀 𝜆 = 𝔼[max(𝑓_ min −𝑌, 0)]
• Used as criteria for selecting next candidate hyperparams for evaluation.
• Often depends on the best hyperparams seen so far in search.
• Controls exploration vs exploitation in search.

Acquisition Function: Expected Improvement
0.3 0.2
𝐸𝐼 𝑥1 > 𝐸𝐼(𝑥2)
𝑥1
𝑥2
1
-1
70%
Current best

Using Acquisition Function
• Expected improvement
[maximining the dashed line] has
two components:
• One is dependent on −𝜇 [solid line]
• The other dependent on uncertainty or
variance 𝑘(𝜆, 𝜆′) [blue line]
• There fore we maximize the
acquisition function wherever:
• Mean, 𝜇, is low, or
• Uncertainty,𝑘(𝜆, 𝜆′), is high.

Part 2: Hands On with Amazon SageMaker AMT

Sagemaker Automatic model tuning

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Sagemaker Automatic model tuning

Semelhante a Sagemaker Automatic model tuning (20)

Último

Último (20)

Sagemaker Automatic model tuning

Notas do Editor