This document discusses machine learning workflows using Kubernetes and KubeFlow. It begins by explaining why Kubernetes is useful for machine learning for its composability, portability, and scalability. It then discusses using Jupyter notebooks for machine learning development workflows. It provides an overview of running machine learning on Kubernetes both with and without KubeFlow. Key components of KubeFlow like Kubeflow Pipelines are explained. Advantages of using KubeFlow on AWS are highlighted. TensorFlow and how AWS optimizes it is covered. The document concludes by discussing characteristics of autonomous vehicle workloads and how Kubeflow on AWS can help.

1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Machine Learning with Kubernetes
Yaniv Donenfeld
AI/ML Solutions
Container Services, AWS
Mike Crute
Sr. Engineer
Container Services, AWS

2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Centerpiece for digital transformation

3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EC2 P3
& P3dn
EC2
C5
FPGAs Greengrass
Elastic
inference
FRAMEWORKS INTERFACES INFRASTRUCTURE
Inferentia
EC2
G4
The Amazon ML stack:
Broadest & deepest set of capabilities

5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Ground Truth Notebooks
Algorithms +
Marketplace
Reinforcement
Learning
Training Optimization Deployment Hosting
ML Frameworks +
Infrastructure EC2 P3
& P3dn
EC2
C5
FPGAs Greengrass
Elastic
inference
FRAMEWORKS INTERFACES INFRASTRUCTURE
Inferentia
EC2
G4
The Amazon ML stack:
Broadest & deepest set of capabilities

6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
R E K O G N I T I O N
I M A G E
P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E XR E K O G N I T I O N
V I D E O
F O R E C A S TT E X T R A C T P E R S O N A L I Z E
VISION SPEECH LANGUAGE CHATBOTS FORECASTING RECOMMENDATIONS
ML Services Amazon
SageMaker Ground Truth Notebooks Algorithms + Marketplace
Reinforcement
Learning
Training Optimization Deployment Hosting
ML Frameworks +
Infrastructure EC2 P3
& P3dn
EC2
C5
FPGAs Greengrass
Elastic
inference
FRAMEWORKS INTERFACES INFRASTRUCTURE
Inferentia
EC2
G4
The Amazon ML stack:
Broadest & deepest set of capabilities

7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Amazon ML stack:
Broadest & deepest set of capabilities
R E K O G N I T I O N
I M A G E
P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E XR E K O G N I T I O N
V I D E O
F O R E C A S TT E X T R A C T P E R S O N A L I Z E
VISION SPEECH LANGUAGE CHATBOTS FORECASTING RECOMMENDATIONS
Amazon
SageMaker Ground Truth Notebooks Algorithms + Marketplace
Reinforcement
Learning
Training Optimization Deployment Hosting
EC2 P3
& P3dn
EC2
C5
FPGAs Greengrass
Elastic
inference
FRAMEWORKS INTERFACES
Inferentia
EC2
G4
INFRASTRUCTURE

8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
“A little less conversation, a little more action, please”
M A C H I N E L E A R N I N G
S T O R A G E
Amazon Redshift
+ Redshift Spectrum
Amazon
QuickSight
Amazon EMR
Hadoop, Spark, Presto,
Pig, Hive…19 total
Amazon
Athena
Amazon
Kinesis
Amazon
Elasticsearch
Service
AWS Glue
A N A L Y T I C S
Amazon S3
Standard-IA
Amazon S3
Standard
Amazon S3
One Zone-IA
Amazon
Glacier
Amazon S3
Intelligent-
Tiering
N E W
Amazon
EBS
Amazon S3
Glacier Deep
Archive
N E W
R E K O G N I T I O N
I M A G E
P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E XR E K O G N I T I O N
V I D E O
F O R E C A S TT E X T R A C T P E R S O N A L I Z E
V I S I O N S P E E C H L A N G U A G E C H A T B O T S F O R E C A S T I N G R E C O M M E N D A T I O N S
Amazon
SageMaker
Ground Truth Notebooks
Algorithms +
Marketplace
Reinforcement
Learning
Training Optimization Deployment Hosting
E C 2 P3
& P3 d n
E C 2
C 5 FPG A s G r e e n g r a s s
E la s t i c
i n f e r e n c e
F R A M E W O R K S I N T E R F A C E S
I n f e r e n t i a
E C 2
G 4
I N F R A S T R U C T U R E

9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
R E K O G N I T I O N
I M A G E
P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E XR E K O G N I T I O N
V I D E O
F O R E C A S TT E X T R A C T P E R S O N A L I Z E
VISION SPEECH LANGUAGE CHATBOTS FORECASTING RECOMMENDATIONS
Amazon
SageMaker Ground Truth Notebooks Algorithms + Marketplace
Reinforcement
Learning
Training Optimization Deployment Hosting
EC2 P3
& P3dn
EC2
C5
FPGAs Greengrass
Elastic
inference
FRAMEWORKS INTERFACES INFRASTRUCTURE
Inferentia
EC2
G4
Machine Learning using Kubernetes

10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ML Frameworks +
Infrastructure EC2 P3
& P3dn
EC2
C5
FPGAs Greengrass
Elastic
inference
FRAMEWORKS INTERFACES INFRASTRUCTURE
Inferentia
EC2
G4
Machine Learning using Kubernetes

11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why Machine Learning on Kubernetes?
Composability Portability Scalability
O N - P R E M I S E S C L O U D
http://www.shutterstock.com/gallery-635827p1.html

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Machine Learning Development Workflow
ML Workflow

14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why Jupyter?
Jupyter provides an easy
on-ramp to build, train,
and deploy ML models
Jupyter Notebook allows you to
create and share documents that
contain live code, equations,
visualizations, and narrative text
Uses include:
• data cleaning and transformation
• numerical simulation
• statistical modeling
• data visualization
• machine learning
• and much more

15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Machine Learning Development Workflow

16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ML on K8s—without KubeFlow
Credits: @aronchik

17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ML on K8s—with KubeFlow
Credits: @aronchik

18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What’s in
KubeFlow?

19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

20. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kubeflow Pipelines
• Invoke EMR/Spark jobs for
preprocessing directly as a
pipeline step
• Invoke SageMaker jobs to
utilize the SageMaker
framework from within
Kubeflow

21. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Advantages of KubeFlow on AWS
EKS cluster provision with
External traffic with
to manage Lustre file system
Centralized and unified K8s logs in
TLS and Auth with and
for your K8s API server endpoint
Detect GPU instance and install

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TensorFlow
Open source library to develop and train ML models
Created by Google Brain team
Can run on desktop, servers, mobiles, edge devices

24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS is the platform of choice to run TensorFlow
of all
TensorFlow
workloads in the
cloud runs on AWS

25. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Train twice as fast
with TensorFlow
65%
Scaling efficiency
with 256 GPUs
S T O C K T E N S O R F L O W
100%
Scaling efficiency
with 256 GPUS
A W S - O P T I M Z E D T E N S O R F L O W

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Autonomous Vehicle Sensor Technology
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
LIDAR
Camera
GPS/HD Maps
Telematics Control Unit
Accelerometer/Gyroscope
DSRC
Radar
Infrared
Ultrasonic

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SAE
LEVEL
NAME CONTROL
MONITORING
VIGILANCE
FALLBACK
HANDOFF
OPERATING
CONDITIONS
0 Unassisted Human
1 Driver Assist
Human and
AD System
Human Human Some
2 Partial Automation AD System Human
Human
(0 warning)
Some
3 Conditional Automation AD System AD System
Human
(adequate warning,
assumed 15 seconds)
Some
4 High Automation AD System AD System AD System Some
5 Full Automation AD System AD System AD System All
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SAE
LEVEL
NAME CONTROL
MONITORING
VIGILANCE
FALLBACK
HANDOFF
OPERATING
CONDITIONS
0 Unassisted Human
1 Driver Assist
Human and
AD System
Human Human Some
2 Partial Automation AD System Human
Human
(0 warning)
Some
3 Conditional Automation AD System AD System
Human
(adequate warning,
assumed 15 seconds)
Some
4 High Automation AD System AD System AD System Some
5 Full Automation AD System AD System AD System All

28. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AV Workload Characteristics
• Distributed, Multi-GPU, Multi-Node Training jobs
• ML Practitioners don’t know/want to deal with infrastructure
• Large datasets on S3 require copying to a staging area on the cluster
• Data should be shared across nodes and re-used across jobs
• Distributed training code can be hard
• Capacity provisioning and smart autoscaling across jobs can be
complex

29. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AV Workload Characteristics
• Distributed, Multi-GPU, Multi-Node Training jobs
• ML Practitioners don’t know/want to deal with infrastructure
Kubeflow on AWS provides an abstraction to setup your
cluster as part of the Kubeflow setup process

30. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AV Workload Characteristics
• Distributed, Multi-GPU, Multi-Node Training jobs
• Large datasets on S3 require copying to a staging area on the cluster
• Data should be shared across nodes and re-used across jobs
• New CSI driver for EFS and FSx Lustre – dynamically provision a
file system using PVC.
• FSx Lustre can shadow an S3 Bucket – no need to copy

31. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AV Workload Characteristics
• Distributed, Multi-GPU, Multi-Node Training jobs
• Distributed training code can be hard
• Use Horovod, a Distributed Training framework for TensorFlow,
Keras, PyTorch, and MXNet

32. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AV Workload Characteristics
• Distributed, Multi-GPU, Multi-Node Training jobs
• Capacity provisioning and smart autoscaling across jobs can be
complex
• Introducing updates to Escalator - a cluster auto scaling
component that is tuned for batch/ML workloads, and EC2 GPU
instance autoscaling

33. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DEMO

34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

35. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AV Workload Characteristics
• Large-scale simulation workloads

36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
-
200,000
400,000
600,000
800,000
1,000,000
1,200,000
Koala Genomics AV Safety
Concurrent CPUs

38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
-
50,000,000
100,000,000
150,000,000
200,000,000
250,000,000
300,000,000
350,000,000
Koala Genomics AV Safety
Total Core Hours / Year

39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Rank / Name Rmax / Rpeak (Petaflops)
1. Summit 143.500 / 200.795
2. Sierra 94.640 / 125.436
3. Sunway Tahihu Light 93.015 / 125.436
4. Tianhe-2A 61.445 / 100.679
5. Piz Daint 21.230 / 27.154
6. Trinity 20.159 / 41.461
7. AI Bridging Cloud Infrastructure 19.880 / 32.577
8. SuperMUCNG 19.477 / 26.874
9. Titan 17.590 / 27.113
10. Sequoia 17.173 / 20.133
TOP500 – Top 10 Supercomputers in Nov 2018

40. We’re helping our AV customers
run at Supercomputer Scale, targeting the
equivalent of one of the Top 10 largest
supercomputers in the world.

41. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenges in setting up containers for ML

42. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS deep learning containers
KEY FEATURES
Customizable
container images
Support for TensorFlow,
Apache MXNet
Single and multi-node
training and inference
Pre-packaged Docker
container images
fully configured
and validated
Best performance
and scalability
without tuning
Works with Amazon EKS,
Amazon ECS,
and Amazon EC2

43. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
16 container images
Training Inference
GPU CPU
Python 2.7 Python 3.6

44. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Getting Started
https://github.com/aws-samples/machine-learning-using-k8s/