Get behind the keyboard for an immersive experience with the newly launched AWS DeepRacer. In this workshop you will get hands-on-experience with reinforcement learning. Developers with no prior machine learning experience will learn new skills and apply their knowledge in a fun and exciting way. You will join a pit crew where you will build and train machine learning models that you can then try out at the MGM Speedway event at the Grand Garden Arena! Please bring your laptop, and start your engines, the race is on!

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DeepRacer –The MGMSpeedway
re:Invent 2018
Speaker Name
Job Title
Company/Org Name
A I M 2 0 6

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
• Introducing AWS DeepRacer ~ 5 min.
• Reinforcement Learning – an introduction ~ 15 min.
• Lab 1 – Pre-Season: Building a Reward Function for AWS DeepRacer ~ 30 min.
• AWS DeepRacer Service and Advanced RL ~ 15 min.
• Lab 2 – Qualifiers: Create an AWS DeepRacer RL Model ~ 22 min.
• AWS DeepRacer, under the hood ~ 10 min.
• AWS DeepRacer League and MGM Speedway ~ 10 min.

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DeepRacerOrigin
How can we put
Reinforcement Learning
in the hands of all
developers? literally

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Robotic autonomous
race car
AnnouncingAWSDeepRacer:Anexcitingwayfordeveloperstogethands-on
experiencewithreinforcementlearning
Racing LeagueVirtual simulator, to train
and experiment

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reinforcement Learning inthebroaderAI context
Reinforcement
Learning
Supervised
Learning
Unsupervised
Learning

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MachineLearningOverview
SUPERVISED UNSUPERVISED REINFORCEMENT

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reinforcement Learning usecases
AUTONOMOUS CARS FINANCIAL TRADING DATACENTER COOLINGFLEET LOGISTICS

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Whatis Reinforcement Learning?

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Howdoes traininghappen? VALUE FUNCTION
POLICY
FUNCTION

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
RLAlgorithms –ValueApproximation
CHALLENGE Explore all (state, action, outcome) combinations in a real
world race to create the value function

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
RLAlgorithms –ValueApproximation
CHALLENGE Explore all (state, action, outcome) combinations in a real
world race to create the value function.
VERDICT Not possible
SOLUTION Approximate the value function using supervised learning.
Use data from exploration to create a model capable of estimating the
likely cumulative reward for all possible actions in a state.
The action with the highest cumulative reward,
is the best action.

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
RLAlgorithms – PolicyApproximation
ALTERNATIVE Try to directly map actions to
states without using a value function.
HOW Use supervised learning to train a policy
function model that returns the best action for
any state.
How do we know this is a good action?
Because it gave the highest reward. Keep
improving our model in the direction of the
highest reward.
FAMOUS EXAMPLE
Dota2 Challenge - OpenAI Five

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DeepRacerConsole

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lab 1– RewardFunctionConstructorsChampionship
OBJECTIVE Console overview and take part in the reward function constructors championship
TIME 30 min.
1. Find the lab content here:
https://github.com/aws-samples/aws-deepracer-workshops
2. Select: Lab 1

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

19. AWS Cloud
AWS
DeepRacer
NAT gateway
VPC
AWS DeepRacer
Models
Simulation
video
Metrics
AWS DeepRacerSimulationArchitecture

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reinforcement LearningAlgorithmsCompared
Value Approximation Policy Approximation
Advantages
More stable performance when it works, and tends to
converge on global optimum
Effective in continuous action spaces, can learn stochastic policies,
and faster convergence
Disadvantages
Difficult to converge if too many (state, action)
combinations, slower convergence in general, and can’t
learn stochastic properties
Typically converges to a local rather than global optimum, high
variance in estimating the gradient adversely affects stability, and
evaluating a policy is generally inefficient
Examples Q-Learning, Deep Q Network, Deep Double Q Network Policy Gradient, Proximal Policy Optimization (PPO)

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DeepRacer NeuralNetworkArchitecture
An overview of the network architecture that AWS DeepRacer uses:
Output

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lab 2–CreateanAWS DeepRacer RLModel
• Objective: Start training your AWS DeepRacer RL Model for the MGM
Speedway
• Time: 22 min.
1. Find the lab content here:
https://github.com/aws-samples/aws-deepracer-workshops
2. Select: Lab 2

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DeepRacerCarSpecifications
CAR 18th scale 4WD with monster truck chassis
CPU Intel Atom™ Processor
MEMORY 4GB RAM
STORAGE 32GB (expandable)
WI-FI 802.11ac
CAMERA 4 MP camera with MJPEG
DRIVE BATTERY 7.4V/1100mAh lithium polymer
COMPUTE BATTERY 13600mAh USB-C PD
SENSORS Integrated accelerometer and gyroscope
PORTS 4x USB-A, 1x USB-C, 1x Micro-USB, 1x HDMI
SOFTWARE Ubuntu OS 16.04.3 LTS, Intel® OpenVINO™
toolkit, ROS Kinetic

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ROS Msg Node
Stored File
ROS
Nodes
Model
Optimizer
Video
M-JPEG
Web Server
Video
Inference
Results
Web
Server
Publisher
Autonomous
Drive
Control
Node
Optimized
Model
Media engine
Camera
Model
Inference
engine
Manual
Drive
Navigation
Node
Servo & Motor
AWS DeepRacerSoftwareArchitecture

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Simulation-to-RealDomain Transfer
SIM-to-REAL CHALLENGE
Train model using simulated images, but race car
using real world images
STRATEGIES
Environment Control
Domain Randomization
Modularity and Abstraction

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DeepRacer League
• The world’s first global, autonomous racing league for developers

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Participatingat theMGMSpeedway
Location MGM Grand Garden Arena
Date and Time
Wednesday Nov 28 | 11:30 am – Midnight
Thursday Nov 29 | 11:30 am – 5:30 pm
Race, Learn, Win Prizes and More at the
2018 re:Invent Speedway!

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DeepRacer NextSteps
AT RE:INVENT
Finish training your model and go to MGM Speedway.
Collect your AWS DeepRacer at the swag counter in the Venetian. Included is a voucher for
shipping, if you choose, that can ONLY be used at the Venetian FedEx.
AFTER RE:INVENT
The account you received today will expire on 12/1/2018.
Sign up for whitelisting access at https://aws.amazon.com/deepracer
Use Amazon SageMaker RL Notebook to continue training your AWS DeepRacer RL
models.
KEEP TRAINING AND KEEP RACING!

34. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Speaker Name
Contact information

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.