Mais conteúdo relacionado Semelhante a Train once, deploy anywhere on the cloud and at the edge with Amazon SageMaker Neo - AIM303 - New York AWS Summit (20) Mais de Amazon Web Services (20) Train once, deploy anywhere on the cloud and at the edge with Amazon SageMaker Neo - AIM303 - New York AWS Summit1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Train once, deploy anywhere on the cloud
and at the edge with Amazon SageMaker
Neo
Vebhhav Singh
Sr. Solutions Architect
AWS
A I M 3 0 3
2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon AI/ML Stack
M L S E R V I C E S
M L F R A M E W O R K S
A I S E R V I C E S
(ML researchers
and academics)
(ML developers
and data scientists)
A m a z o n
S a g e M a k e r
G R O U N D T R U T H A L G O R I T H M S
N O T E B O O K S
M A R K E T P L A C E
U N S U P E R V I S E D
L E A R N I N G
S U P E R V I S E D
L E A R N I N G
R E I N F O R C E M E N T
L E A R N I N G
O P T I M I Z A T I O N
( N E O )
T R A I N I N G
H O S T I N G
D E P L O Y M E N T
Vision Speech Language Chatbots &
Contact Centers
Verticals
A m a z o n
R e k o g n i t i o n
I m a g e
A m a z o n
R e k o g n i t i o n
V i d e o
A m a z o n
P o l l y
A m a z o n
T r a n s c r i b e
A m a z o n
T r a n s l a t e
A m a z o n
C o m p r e h e n d A m a z o n
L e x
A m a z o n
F o r e c a s t
A m a z o n
T e x t r a c t
A m a z o n
P e r s o n a l i
(Applicaion
Developers)
3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Deployment of ML models is complex
Repeat for every model and every change in the model
BYO
AWS
Build with your
own algorithms
Build with built-in algorithms
from AWS
Train with
TensorFlow,
MXNet ,
or PyTorch
Optimize
your models
Deploy to
the cloud
Deploy to
the edge
A B
A/B test
4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The majority of the cost and
complexity of ML in
production is due to Inference
Inference (Prediction)
90%
Training
10%
5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Problem: Not all models are skinny
Models that are accurate tend to
be big and slow
Models are chained to the
framework in which they were
trainedS
L
M
Accuracy
Performance
6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Problem: Not all targets are easy
Need enormous expertise …
Application Development: Cloud-Native or Embedded System
Machine Learning: Model Training and Parameter Tuning
Performance Tuning: Troubleshooting and Optimization
Frameworks: TensorFlow or MXNet or PyTorch or Chainer
Hardware: Cloud Server or Edge Device
Computer Architecture: x86 or RISC or GPU or FPGA or ASIC
…and endless time
7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Problem: Not every path is a catwalk
8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What if you could train your model once and run it
anywhere, in the cloud or at the edge, with twice the
speed and no loss in accuracy?
9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SageMaker Neo
10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Increase inference performance by 2x
Reduce runtime footprint by 100x
11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SageMaker Neo
K e y f e a t u r e s
Machine Learning Compiler for CPU or GPU
Compact Open Source Runtime
https://aws.amazon.com/sagemaker/neo
12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Neo bridges Amazon SageMaker & AWS IoT Greengrass
Data labeling &
pre-built
notebooks
for common
problems
Model and
algorithm
marketplace &
built-in, high-
performance
algorithms
One-click
training on the
highest
performing
infrastructure
One-click model
optimization
One-click
deployment
Improves performance on
selected hardware
Extends AWS IoT
onto your devices
NeoAmazon SageMaker AWS IoT Greengrass
ML models train once run anywhere
13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Neo delivers compilation as a service
Parses
model
Optimizes
tensors
Generates
code
Optimizes
graph
Convert a TensorFlow,
MXNet, PyTorch, or
XGBoost model into a
common format
Detect patterns in the
ML model structure to
reduce the execution
time
Detect patterns in the shape
of input data to allocate
memory efficiently
Use a low-level compiler to
generate machine code for
each target
No additional cost for Amazon SageMaker users
14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Neo delivers compilation as a service
Parses
model
Optimizes
tensors
Generates
code
Optimizes
graph
Graph pruning
Layer fusion
Constant folding
Layout transforms
Nested parallelism
Tiling
Tensorization
Auto-tuning
Uses Treelite and Apache TVM for model optimization
15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Neo delivers a compact runtime
Dispatches
model
Partitions
graph
Matches model with
execution backend
Sends subgraph to
suitable accelerator
Framework Size
MXNet 450 MB
TensorFlow 660 MB
PyTorch 1000 MB
Neo 1 MB
Open-source software enables device-specific customization
16. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Steps for sample deployment
- Get an existing model
- Compile model for new hardware
- Copy the artifacts to new hardware
- Deploy the Neo runtime
- Run the inference
18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Neo: Open-source project
19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Q&A
20. Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Vebhhav Singh
vebhhavs@amazon.com