This presentation was given at AWS Builders' Days in December 2018

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Julien Simon
Principal Technical Evangelist, AI & Machine Learning,
AWS
@julsimon
From Notebook to Production
with Amazon SageMaker

2. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Put Machine Learning in the hands
of every developer and data scientist
Our mission

3. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Application
Services
Platform
Services
Frameworks
& Infrastructure
API-driven services: Vision, Language & Speech Services, Chatbots
AWS ML Stack
Deploy machine learning models with high-performance machine learning algorithms,
broad framework support, and one-click training, tuning, and inference.
Develop sophisticated models with any framework, create managed, auto-scaling
clusters of GPUs for large scale training, or run prediction
on trained models.

4. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Application
Services
Platform
Services
Frameworks
& Infrastructure
API-driven services: Vision, Language & Speech Services, Chatbots
AWS ML Stack
Deploy machine learning models with high-performance machine learning algorithms,
broad framework support, and one-click training, tuning, and inference.
Develop sophisticated models with any framework, create managed, auto-scaling
clusters of GPUs for large scale training, or run prediction
on trained models.

5. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Data Visualization &
Analysis
Business Problem
ML problem framing Data Collection
Data Integration
Data Preparation &
Cleaning
Feature Engineering
Model Training &
Parameter Tuning
Model Evaluation
Are Business
Goals met?
Model Deployment
Monitoring &
Debugging
YesNo
DataAugmentation
Feature
Augmentation
The Machine Learning Process
Re-training
Predictions

6. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Amazon SageMaker
Pre-built
notebooks for
common
problems
K-Means Clustering
Principal Component Analysis
Neural Topic Modelling
FactorizationMachines
Linear Learner
XGBoost
Latent DirichletAllocation
Image Classification
Seq2Seq,
And more!
ALGORITHMS
Apache MXNet, Chainer
TensorFlow, PyTorch, scikit-learn
FRAMEWORKS Set up and manage
environments for training
Train and tune
model (trial and
error)
Deploy model
in production
Scale and manage the
production environment
Built-in, high-
performance
algorithms
Build
Git integration
Elastic inference

7. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Git integration

8. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Elastic Inference
https://aws.amazon.com/blogs/aws/amazon-elastic-inference-gpu-powered-deep-learning-inference-acceleration/

9. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Amazon SageMaker
Pre-built
notebooks for
common
problems
K-Means Clustering
Principal Component Analysis
Neural Topic Modelling
FactorizationMachines
Linear Learner
XGBoost
Latent DirichletAllocation
Image Classification
Seq2Seq,
And more!
ALGORITHMS
Apache MXNet, Chainer
TensorFlow, PyTorch, scikit-learn
FRAMEWORKS Set up and manage
environments for training
Train and tune
model (trial and
error)
Deploy model
in production
Scale and manage the
production environment
Built-in, high-
performance
algorithms
Build
New built-in algorithms
scikit-learn environment
Model marketplace
Search

10. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Search training jobs

11. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Machine Learning Marketplace

12. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Amazon SageMaker
Pre-built
notebooks for
common
problems
Built-in, high-
performance
algorithms
One-click
training
Hyperparameter
optimization
Train
Deploy model
in production
Scale and manage the
production
environment
P3DN, C5N
TensorFlow on 256 GPUs
Resume HPO tuning job
Build

13. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Amazon SageMaker
Fully managed
hosting with auto-
scaling
One-click
deployment
Pre-built
notebooks for
common
problems
Built-in, high-
performance
algorithms
One-click
training
Hyperparameter
optimization
Deploy
Model compilation
Elastic inference
Inference pipelines
TrainBuild

14. Working with Amazon SageMaker

15. The Amazon SageMaker API
• Python SDK orchestrating all Amazon SageMaker activity
• High-level objects for algorithm selection, training, deploying,
automatic model tuning, etc.
• Spark SDK (Python & Scala)
• AWS CLI: ‘aws sagemaker’
• AWS SDK: boto3, etc.

16. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Model Training (on EC2)
Model Hosting (on EC2)
Trainingdata
Modelartifacts
Training code Helper code
Helper codeInference code
GroundTruth
Client application
Inference code
Training code
Inference requestInference response
Inference Endpoint

17. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Training code
Factorization Machines
Linear Learner
Principal Component Analysis
K-Means Clustering
XGBoost
And more
Built-in Algorithms Bring Your Own ContainerBring Your Own Script
Model options

18. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Built-in algorithms
orange: supervised, yellow: unsupervised
Linear Learner: regression, classification Image Classification: Deep Learning (ResNet)
Factorization Machines: regression, classification,
recommendation
Object Detection (SSD): Deep Learning
(VGG or ResNet)
K-Nearest Neighbors: non-parametric regression and
classification
Neural Topic Model: topic modeling
XGBoost: regression, classification, ranking
https://github.com/dmlc/xgboost
Latent Dirichlet Allocation: topic modeling (mostly)
K-Means: clustering Blazing Text: GPU-based Word2Vec,
and text classification
Principal Component Analysis: dimensionality
reduction
Sequence to Sequence: machine translation, speech
to text and more
Random Cut Forest: anomaly detection DeepAR: time-series forecasting (RNN)
Object2Vec: general-purpose embedding IP Insights: usage patterns for IP addresses
Semantic Segmentation: Deep Learning

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Blazing Text
https://dl.acm.org/citation.cfm?id=3146354

20. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Demo:
Text Classification with BlazingText
https://github.com/awslabs/amazon-sagemaker-
examples/tree/master/introduction_to_amazon_algorithms/blazingtext_text_classification_dbpedia

21. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Optimizing TensorFlow
https://aws.amazon.com/blogs/machine-learning/faster-training-with-
optimized-tensorflow-1-6-on-amazon-ec2-c5-and-p3-instances/ (March
2018)
Training a ResNet-50 benchmark with
the synthetic ImageNet dataset using
our optimized build of TensorFlow
1.11 on a c5.18xlarge instance type is
11x faster than training on the stock
binaries.
https://aws.amazon.com/about-aws/whats-
new/2018/10/chainer4-4_theano_1-0-
2_launch_deep_learning_ami/ (October 2018)

22. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Demo:
Elastic Inference with TensorFlow
https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-
sdk/tensorflow_iris_dnn_classifier_using_estimators/

23. Automatic Model Tuning

24. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Automatic Model Tuning
Finding the optimal set of hyper parameters
1. Manual Search (”I know what I’m doing”)
2. Grid Search (“X marks the spot”)
• Typically training hundreds of models
• Slow and expensive
3. Random Search (“Spray and pray”)
• Works better and faster than Grid Search
• But… but… but… it’s random!
4. HPO: use Machine Learning
• Training fewer models
• Gaussian Process Regression and Bayesian Optimization
• You can now resume from a previous tuning job

25. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Demo: HPO with TensorFlow
https://github.com/awslabs/amazon-sagemaker-examples/tree/master/hyperparameter_tuning/tensorflow_mnist
CONV1
POOL1
CONV2
POOL2
DENSE1
DROPOUT1
OUTPUT
Class
probabilities
INPUT

26. Model Compilation

27. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Optimizing for the underlying hardware
https://aws.amazon.com/blogs/aws/amazon-sagemaker-neo-train-your-machine-learning-models-once-run-them-anywhere/
• Train once, run anywhere
• Frameworks and algorithms
• TensorFlow, Apache MXNet, PyTorch, ONNX, and XGBoost
• Hardware architectures
• ARM, Intel, and NVIDIA starting today
• Cadence, Qualcomm, and Xilinx hardware coming soon
• Amazon SageMaker Neo will be released as open source enabling hardware
vendors to customize it for their processors and devices.

28. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Demo:
Compiling ResNet-50 for the Raspberry Pi
Configure the compilation job
{
"RoleArn":$ROLE_ARN,
"InputConfig": {
"S3Uri":"s3://jsimon-neo/model.tar.gz",
"DataInputConfig": "{"data": [1, 3, 224, 224]}",
"Framework": "MXNET"
},
"OutputConfig": {
"S3OutputLocation": "s3://jsimon-neo/",
"TargetDevice": "rasp3b"
},
"StoppingCondition": {
"MaxRuntimeInSeconds": 300
}
}
Compile the model
$ aws sagemaker create-compilation-job
--cli-input-json file://config.json
--compilation-job-name resnet50-mxnet-pi
$ aws s3 cp s3://jsimon-neo/model-
rasp3b.tar.gz .
$ gtar tfz model-rasp3b.tar.gz
compiled.params
compiled_model.json
compiled.so
Predict with the compiled model
from dlr import DLRModel
model = DLRModel('resnet50', input_shape,
output_shape, device)
out = model.run(input_data)

29. Inference Pipelines

30. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Inference Pipelines
• Linear sequence of 2-5 containers that process inference requests
• Feature engineering with scikit-learn or SparkML (on AWS Glue or Amazon EMR)
• Predict with built-in or custom containers
• The sequence is deployed as a a single model
• Useful to preprocess, predict, and post-process
• Available for real-time prediction and batch transform

31. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Demo:
Inference with scikit-learn and linear learner
https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-
sdk/scikit_learn_inference_pipeline/

32. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Amazon SageMaker
Fully managed
hosting with auto-
scaling
One-click
deployment
Pre-built
notebooks for
common
problems
Built-in, high-
performance
algorithms
One-click
training
Hyperparameter
optimization
Build Train Deploy

33. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Selected Amazon SageMaker customers

34. Automatic Model Tuning

35. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Resources
http://aws.amazon.com/free
https://ml.aws
https://aws.amazon.com/sagemaker
https://github.com/aws/sagemaker-python-sdk
https://github.com/aws/sagemaker-spark
https://github.com/awslabs/amazon-sagemaker-examples
https://gitlab.com/juliensimon/ent321 SageMaker workshop at AWS re:Invent 2018
https://medium.com/@julsimon
https://gitlab.com/juliensimon/dlnotebooks

36. Thank you!
Julien Simon
Principal Technical Evangelist, AI and Machine Learning
@julsimon