1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Injae Kwak - AWS Solutions Architect
3rd July 2019
Building an immersive, interactive customer
experience using artificial intelligence and
augmented reality

2. Agenda
• Primer on XR (Augmented Reality/Virtual Reality)
• Building interactive smart applications
• Combining AI/ML and AR
• Getting started with AWS services
• Variety showcase of demos

3. To understand Augmented Reality
We must go back..

4. Great Scott!

5. The real world Fully immersive
Digital information
layered over the
real world
Virtual objects
interacting with real
world
Blend of digital and
real world info to
enhance reality
(we are here!)

6. Web ImmersiveMobile Cloud
Evolution of User Interfaces

7. How customers are using XR
Instructional design
Brand engagement
Data visualisation
Simulation
Digital Twins
Virtual tours
Digital signage Customer Service
agents & chatbots

8. XR customers across industries

9. What are some of the challenges?

10. Amazon Sumerian

11. Leverage rich library of assets

12. Entities Panel
Canvas
Assets Panel
State Graph
WebVR Mode
Components Panel

13. Demo: Web based AR

14. Building interactive experience through AI/ML

15. M L F R A M E W O R K S &
I N F R A S T R U C T U R E
The Amazon AI/ML Stack
A I S E R V I C E S
A M A Z O N
R E K O G N I T I O N
I M A G E
A M A Z O N
P O L L Y
A M A Z O N
T R A N S C R I B E
A M A Z O N
T R A N S L A T E
A M A Z O N
C O M P R E H E N D
C O M P R E H E N D
M E D I C A L
A M A Z O N
L E X
A M A Z O N
R E K O G N I T I O N
V I D E O
Vision Speech Chatbots
A M A Z O N S A G E M A K E R
B U I L D T R A I N
A M A Z O N
F O R E C A S T
A M A Z O N
T E X T R A C T
A M A Z O N
P E R S O N A L I Z E
D E P L O Y
Pre-built algorithms & notebooks
Data labeling (G R O U N D T R U T H )
One-click model training & tuning
Optimisation ( N E O )
One-click deployment & hosting
M L S E R V I C E S
F r a m e w o r k s I n t e r f a c e s I n f r a s t r u c t u r e
E C 2 P 3
& P 3 d n
E C 2 C 5 F P G A s G R E E N G R A S S E L A S T I C
I N F E R E N C E
Models without training data (REINFORCEMENT LEARNING)Algorithms & models ( A W S M A R K E T P L A C E )
Language Forecasting Recommendations
NEW
NEWNEW
NEW
NEW
NEW NEWNEWNEW

16. M L F R A M E W O R K S &
I N F R A S T R U C T U R E
The Amazon AI/ML Stack
A I S E R V I C E S
A M A Z O N
R E K O G N I T I O N
I M A G E
A M A Z O N
P O L L Y
A M A Z O N
T R A N S C R I B E
A M A Z O N
T R A N S L A T E
A M A Z O N
L E X
A M A Z O N
R E K O G N I T I O N
V I D E O
Vision Speech Chatbots
A M A Z O N S A G E M A K E R
B U I L D T R A I N
A M A Z O N
F O R E C A S T
A M A Z O N
T E X T R A C T
A M A Z O N
P E R S O N A L I Z E
D E P L O Y
Pre-built algorithms & notebooks
Data labeling (G R O U N D T R U T H )
One-click model training & tuning
Optimization ( N E O )
One-click deployment & hosting
M L S E R V I C E S
F r a m e w o r k s I n t e r f a c e s I n f r a s t r u c t u r e
E C 2 P 3
& P 3 d n
E C 2 C 5 F P G A s G R E E N G R A S S E L A S T I C
I N F E R E N C E
Models without training data (REINFORCEMENT LEARNING)Algorithms & models ( A W S M A R K E T P L A C E )
Language Forecasting Recommendations
NEW
NEWNEW
NEW
NEW
Pre-trained AI services that
require no ML skills or training
Easily add intelligence to your
existing apps and workflows
Quality and accuracy from
continuously-learning APIs
A M A Z O N
C O M P R E H E N D
C O M P R E H E N D
M E D I C A L
NEW NEWNEWNEW

17. M L F R A M E W O R K S &
I N F R A S T R U C T U R E
The Amazon AI/ML Stack
A I S E R V I C E S
A M A Z O N
R E K O G N I T I O N
I M A G E
A M A Z O N
P O L L Y
A M A Z O N
T R A N S C R I B E
A M A Z O N
T R A N S L A T E
A M A Z O N
L E X
A M A Z O N
R E K O G N I T I O N
V I D E O
Vision Speech Chatbots
A M A Z O N S A G E M A K E R
B U I L D T R A I N
A M A Z O N
F O R E C A S T
A M A Z O N
T E X T R A C T
A M A Z O N
P E R S O N A L I Z E
D E P L O Y
Pre-built algorithms & notebooks
Data labeling (G R O U N D T R U T H )
One-click model training & tuning
Optimization ( N E O )
One-click deployment & hosting
M L S E R V I C E S
F r a m e w o r k s I n t e r f a c e s I n f r a s t r u c t u r e
E C 2 P 3
& P 3 d n
E C 2 C 5 F P G A s G R E E N G R A S S E L A S T I C
I N F E R E N C E
Models without training data (REINFORCEMENT LEARNING)Algorithms & models ( A W S M A R K E T P L A C E )
Language Forecasting Recommendations
NEW
NEWNEW
NEW
NEW
A M A Z O N
R E K O G N I T I O N
I M A G E
Object, scene, and activity detection
Facial recognition
Facial analysis
Pathing people movement
Unsafe content detection
Celebrity detection
Text extraction
A M A Z O N
C O M P R E H E N D
C O M P R E H E N D
M E D I C A L
NEW NEWNEWNEW

18. Demo: Utilising AI through APIs

19. Application architecture
Mobile client
Amazon Cognito
Amazon Rekognition
Authentication
Model Inference
User App
Object Detection App

20. Customers using Amazon Rekognition

21. M L F R A M E W O R K S &
I N F R A S T R U C T U R E
The Amazon AI/ML Stack
A I S E R V I C E S
R E K O G N I T I O N
I M A G E
P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D
C O M P R E H E N D
M E D I C A L
L E XR E K O G N I T I O N
V I D E O
Vision Speech Chatbots
A M A Z O N S A G E M A K E R
B U I L D T R A I N
F O R E C A S TT E X T R A C T P E R S O N A L I S E
D E P L O Y
Pre-built algorithms & notebooks
Data labeling (G R O U N D T R U T H )
One-click model training & tuning
Optimization ( N E O )
One-click deployment & hosting
M L S E R V I C E S
F r a m e w o r k s I n t e r f a c e s I n f r a s t r u c t u r e
E C 2 P 3
& P 3 d n
E C 2 C 5 F P G A s G R E E N G R A S S E L A S T I C
I N F E R E N C E
Models without training data (REINFORCEMENT LEARNING)Algorithms & models ( A W S M A R K E T P L A C E )
Language Forecasting Recommendations
NEW NEWNEW
NEW
NEW
NEWNEW
NEW
NEW

22. Using a custom machine learning model
Use Case: Sign Language to English Translation
Handshape
Position
Movement
Non-Manual Markers
Context

23. Demo: Translating Sign Language

24. Application architecture
Mobile client
Amazon Cognito
Amazon Rekognition
Authentication
Model Inference
User App
Object Detection App
Mobile client
Amazon Cognito
Amazon SageMaker
Authentication
Model Inference
User App
Sign Language Translation App

25. What about combining both together?
+
Augmented
Reality
Artificial
Intelligence
= ? ? ?

26. Demo: Augmented Reality Concierge

27. So how do I get started?

28. Introducing AWS Amplify
XR
Authentication
Analytics
Pub/Sub
APIs

29. Dream Big
• Drive innovation through imagination
• Disrupt customer experience using emerging technologies
Start Small
• Add interactive experiences through AWS AI/ML and Amazon Sumerian
• Integrate rich functionalities through AWS Amplify
Build Fast
• Links to tutorials and labs @ next page
• Links to download demos @ next page

30. Additional Resources
https://bit.ly/2TUJ2acAmazon Sumerian Tutorials
Github Lab – Starter React PWA https://bit.ly/2GkrT4b
Sign Language Translation iOS app (testflight) https://apple.co/2uUmbh8
Object Detection iOS app (testflight) https://apple.co/2D65aVJ
AR Concierge Android app (google playstore) https://bit.ly/2FZzWjA