This document discusses deploying deep learning models in ONNX format with the Apache MXNet Model Server. It describes how ONNX serves as a common format that can be exported from many frameworks and imported into MXNet. It also outlines how the MXNet Model Server provides an API and containerization for serving models, and how it can be deployed in serverless environments on AWS Fargate for scalable inference.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deploying Your ONNX Deep
Learning Models with Apache MXNet
Model Server
Girish Patil
Deep Learning Solutions
Architect
A I M 4 1 3
Steffen Rochel
Head of Engineering
Deep Engines

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Performance
Availability
Networking
Monitoring
Modeldecoupling
Crossframework
Crossplatform
The
undifferentiated
heavy lifting of
model serving

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture
Back

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Trained
network
Model
signature
Custom
code
Auxiliary
assets
Modelarchive
Model ExportCLI
Model archive
Back

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
O(n2)
Pairs
MXNet
Caffe2
PyTorch
Tensorflow
CNTKCoreML
TensorRT
NGraph
SNPE
Open Neural Network eXchange—Overview
Many frameworks
Many platforms
ONNX: Common IR
Neural Network format
Open source
Growing support

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
# Import into MXNet (from MXNet 1.2)
sym, arg_params, aux_params = onnx_mxnet.import_model('model.onnx’)
# create module
mod = mx.mod.Module(symbol=sym, data_names=['input_0'], label_names=None)
mod.bind(for_training=False, data_shapes=[('input_0', input_img.shape)])
mod.set_params(arg_params=arg_params, aux_params=aux_params)
ONNX—Usage example
Back

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ONNX—Usage example
# Synthetic input for tracing
x = Variable(torch.randn(batch_size, 1, 224, 224), requires_grad=True)
# Export the model
torch_out = torch.onnx.export(model, x, "model.onnx")

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
REST and OpenAPI
Back

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Client code generation

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MMS
docker
image
Pull or build
run
Containerization
Containercluster
MMS Container
MMS ContainerMMScontainer
MXNet ModelServer
MXNet NGINX
Load
balance
r

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A practical scenario

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MMS Server
Client
Pre-processing & post processing
Request
Response
Pre-
processing
Post
processing
Model
Keeps clients simple!

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pre-processing sample code

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Post processing sample code

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
+
MXNet Model Server AWS Fargate
Serverless model serving

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Serverless model serving architecture
VPC
ECS cluster and service
Fargate task(s) Load
balancer
Internet
CloudWatch
MMS container
Model
Server

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker:
https://aws.amazon.com/sagemaker/
Using Apache MXNet with Amazon SageMaker:
https://docs.aws.amazon.com/sagemaker/
latest/dg/mxnet.html
Contact: mxnet-info@amazon.com

25. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Girish Patil Steffen Rochel
girpatil@amazon.com steroche@amazon.com

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.