This document discusses deep learning model serving. It introduces Apache MXNet as a deep learning framework and describes how MXNet Model Server can be used to deploy trained models and serve predictions. Key aspects covered include model archives, containerization of models for deployment, operational metrics, and challenges for further improving model serving platforms.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Model Serving for Deep Learning
Hagay Lupesko, Amazon AI
©2018 Amazon Web Services, Inc. or its affiliates, All rights reserved

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deep Learning is a Big Deal
It has a growing impact on our lives
Personalization Logistics Voice
Autonomous
Vehicles

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deep Learning is a Big Deal
It’s able to do better than humans (and ML)

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Model
Model Server
Mobile
Desktop
IoT
Internet
So what does a deployed model looks like?

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Performance
Availability
Networking
Monitoring
Model Decoupling
Cross Framework
Cross Platform
The Undifferentiated
Heavy Lifting of
Model Serving

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Apache MXNet - Background
• Apache (incubating) open source project
• Framework for building and training DNNs
• Created by academia (CMU and UW)
• Adopted by AWS as DNN framework of
choice, Nov 2016 http://mxnet.io

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Apache MXNet - Highlights
• Imperative, Symbolic and Dynamic APIs
• APIs in Python, Scala, C, C++, R (and more)
• Examples and tutorials
Ease of Use
• Optimized for CPU, GPU, ARM (and more)
• Highly scalable distributed training
• Quantization, Sparse, NCCL, and more…
Performance
• Train on cloud, predict on edge
• Model serving framework
• ONNX support
Portability

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
It’s Demo Time!

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Model Archive
REST and OpenAPI
Containerization
ONNX Support Operational Metrics

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Trained
Network
Model
Signature
Custom
Code
Auxiliary
Assets
Model Archive
Model Export CLI
Model Archive

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
MMS
Docker
Image
Docker Run
Containerization
Lightweight virtualization, isolation, runs anywhere
Container Cluster
MMS Container
MMS ContainerMMS Container
MXNet NGINX
MXNet Model Server
Load
Balancer

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Requests
• Latencies
• Resources
Metrics
• Model Name
• Host Name
Dimensions
• Log / CSV
• AWS CloudWatch
Target
Operational Metrics
Back

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Performance
• Batching
• Caching
• JIT
Compilation
• Custom code
• Quantization
Platform
• New players
• ONNX
• Plugins
Adoption
• Ease of use
• Internal
Amazon dev
tools
• Industry
partners
Challenges Ahead

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Try it out, Star and Contribute!
github.com/awslabs/mxnet-model-server