In this workshop, we build and deploy an end-to-end analytics solution for monitoring the video streaming experience. We integrate an open source video player with Amazon Kinesis Data Streams to capture events in real time. We explore the data available for capture and a variety of use cases: from generating alerts on poor experience to content recommendations based on user behavior. We also show you how this real-time data can be archived in a data lake and further used to generate reports of aggregate performance and experience across a number of dimensions.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Build an AWS Analytics Solution to
Monitor the Video Streaming Experience
M A E 3 0 9
Alastair Cousins
Solutions Architect
Sydney, Australia
Jaiganesh Girinathan
Solutions Architect
Mumbai, India
Jarrod Spiga
Solutions Architect
Sydney, Australia

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Changes in viewers habits are accelerating
3
Customers expect new content
options available anytime,
anywhere, on any device, and in
the highest possible quality
• OTT growing at almost 10x the
pace of pay TV. Within five years
could make up 1/3 of the
market (from 15% today)
• Increase in choice: 400% more
content choices per person from
2007 to 2017
Source:
ABI Research, Over the Top (OTT) and Multiscreen Video Services 2017
Smart Screen News, Avid CEO: ‘Massive Explosion’ of Content Has Created New Challenges for Media Companies, Jan 2018

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What you will learn today
• How to measure the Quality of Experience with streaming video
• How to capture video player events using the open source video.js
video player
• How to build an event pipeline, delivering analytics data into an AWS
data lake
• How to build reports from the data lake using serverless architecture
• How to capture real-time metrics and use this to enhance the viewer
experience

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Logistics and resources
• You will need
• A laptop, feel free to work in groups
• AWS account ($25 credit on us)
• Firefox or Chrome browser
• Text editor or an IDE to look at code
• Lab guide:
https://mediaqos-reinvent2018-workshop.s3.amazonaws.com/index.html
• Working in us-west-2 region
• AWS CloudFormation template deployment takes 15-20 minutes
Do this now

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Server side metrics
CDN logs
Content origin
Streaming video

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Client side metrics
CDN logs
Content origin
Streaming video

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Streaming video experience
Quality
Jump

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Streaming video experience
Player Events

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Streaming video experience
Play event Direction
event
Stop
event

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Buffering experience
Buffer
event

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Buffering experience
Buffer
event

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Seeking experience
Seek
event
Play
event

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sources
Modern data architecture
Insights to enhance viewer engagement, personalization, monetization
Viewer data
Engage & personalize
IoT sensors
Improve UX/ QoS
Social media
Recommend
Monetize
Video
Ingest/
collect
Infer/
insights
Store
content/data lake
Process/analyze
1 4
0 9
5

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture—Ingest pipeline
CloudFront
access logs
Player events
Push to
Kinesis Data
Firehose
Video views
Obtain
credentials
S3 PUT event
Events

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture—Analytics
Lambda
Function
Reporting layer
Real-time layer
Schema
definition
Hive
metastore
schema
Player event data SQL queries
Slidingwindow
analysis
Post updates Update item

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture—Real-time player updates
Lambda
Function
Table of current
viewers

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture—Ingest pipeline
CloudFront
access logs
Player events
Push to
Kinesis Firehose
Video views
S3 PUT event
Data lake
Real-time
processing
Obtain
credentials

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our example client: Video.js
• JavaScript/HTML5 video player
• Open source (Apache License)
• Plugin architecture
QoS plugin
Plugin architecture
HLS plugin
DASH plugin
https://github.com/videojs/video.js

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Firehose—How it works
• Zero administration and seamless elasticity
• Direct-to-data store integration
• Serverless, continuous data transformations

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Player scope and variations

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Real-time layer
Architecture—Analytics
Lambda
Function
Reporting layer

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Glue
• The schema for our data is defined
in an AWS Glue schema
• AWS Glue publishes a Hive
metastore catalog which is used
by Amazon Athena to understand
the schema
• Partitions are periodically created
by AWS Glue for use by Athena to
efficiently query the data set

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Athena and Amazon QuickSight
• Amazon QuickSight provides a visual
interface allowing business users to
easily generate charts and
dashboards
• Our implementation uses Athena to
query the data set stored in the
Amazon S3 backed data lake
• End users generate reports,
leveraging the schema provided by
AWS Glue

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture—Real-time player updates
Lambda
Function
Table of current
viewers

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Analytics—How it works

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS AppSync & GraphQL

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Real-time events
• Enhancing customer experience
• Active users on the platform right now
• Current viewers for each item
• Popular items: What is trending?
• Operational monitoring
• Availability monitoring
• User experience by geography or item
• CDN selection

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS AppSync data flow—On startup
Lambda
Function
Get item
Query for
current metrics
Table of current
viewers

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS AppSync data flow—New data
Lambda
Function
Viewer count
update
Update Item
Updated metric
pushed to player
Player events
Table of current
viewers

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Clean up
• Make sure you run through the clean up lab before leaving!
• Three steps:
• Empty Amazon S3 Buckets for logs and application
• Delete AWS CloudFormation Stack deployed today
• Unsubscribe from Amazon QuickSight
• Your feedback is really important to us!
• Further questions?
• acousins@amazon.com

44. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.