Companies have ever-growing media libraries, making them increasingly difficult to index and search. In this session, we describe how to maintain your library by using Amazon Rekognition, Amazon Transcribe, and Amazon Comprehend to perform automatic metadata extraction from image, video, and audio files. We show you how to then use this metadata to build a serverless media library that can be filtered by image tags, celebrities, and more.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Create a Serverless Searchable
Media Library
Liam Morrison
Principal Solutions Architect
Amazon Web Services
A I M 3 4 2
Gene Ting
Solutions Architect
Amazon Web Services

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ingest Store Analyze Deliver
Petabytes of
images
and video assets
Centralized
storage
& global registry
Metadata enrichment
through deep learning
Enhanced value
and search experience
Feedback
loop
Media Workflow Enrichment

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Amazon Machine Learning (Amazon ML) Stack
ML services Amazon SageMaker AWS DeepLens
A m a z o n
R e k o g n i t i o n
I m a g e
A m a z o n
P o l l y
A m a z o n
T r a n s c r i b e
A m a z o n
T r a n s l a t e
A m a z o n
C o m p r e h e n d
A m a z o n
L e x
A m a z o n
R e k o g n i t i o n
V i d e o
Vision Speech Language Chatbots &
contact centers
AI services
ML frameworks

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Facial analysis Face recognition
Text in image
Deep learning-based image analysis service
Unsafe image detection
Object & scene detection
Celebrity recognition
Amazon Rekognition Video

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition Video
Object & activity detection Face detection &
recognition
Real-time live stream
Deep learning-based video analysis service
Unsafe video detectionCelebrity recognition
Pathing

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Transcribe
Automatic speech recognition
Time stamps and
confidence scores
Support for both
regular and telephony
audio
Punctuation
§
Amazon S3
integration
Hello
/Hola
English and Spanish
with more to come

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Comprehend
Discover insights and relationships in text
Sentiment Entities Languages Key phrases Topic modeling
POWERED BY DEEP
LEARNING

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Machine learning services
Language
Amazon Comprehend,
Amazon Transcribe
Vision
Amazon Rekognition Image &
Video
Media Analysis Solution starter kits
Go To aws.amazon.com > answers > media-entertainment > Media-Analysis-Solution
Automated metadata generation
Label & face detection
Celebrity detection
Face search
Person in picture tracking
Subtitling
Context
Key entity & phrase detection
Other Services
such as Amazon Translate Future starter kits
Automatically provision the services necessary for building common media use cases on AWS
Media Analysis Solution Interface

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Media Analysis Solution
Serverless
website,
built on
Amplify
Authentication
with Amazon
Cognito User
Pools
Dynamic
website
interactions
Media is
uploaded to S3
S3 event triggers
Lambda function,
which parses the
event details and
starts the state
machine
AWS Step Functions
coordinates the
processing of the
media using AI
services
Processes the video
and images for visual
information (objects,
people, faces,
celebrities, etc.)
Audio is extracted
from the video files
for transcription
Audio is transcribed
with timecodes to be
used for captions
The transcribed
content is analyzed
for key phrases and
entities
Results are indexed
in the Elasticsearch
cluster
Retrieves results from
ElasticSearch / S3

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker
Fully managed
hosting with auto-
scaling
One-click
deployment
Pre-built
notebooks for
common
problems
Built-in, high-
performance
algorithms
One-click
training
Hyperparameter
optimization
Build Train Deploy

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Media Analysis Solution State Machine
1. Evaluates if it is
audio, video, or
image
2. Passes parameters to
AI/ML services
(Amazon
Rekognition,
Amazon Transcribe,
and Amazon
Comprehend)
3. Validates results
from AI/ML services
and passes them in
to Amazon
Elasticsearch Service

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Extending the Media Analysis Solution

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Video Versus Still-frame Inference

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Extracting Frames from Video for Analysis

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Extending the Media Analysis Solution for Video

18. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Liam Morrison
Lmorri@amazon.com
Gene Ting
Geneting@amazon.com

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.