AWS Immersion Day - Image Data Insights & Analytics Specialist Session - June 2019

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Public Sector
June 2019
Using AI and ML services on video streams
Immersion day

Why Machine Learning for Media?
Increase
Engagement and
Content Discovery
Accelerate
information through
metadata
Launch new content
or services

Agenda
• Quick Service Overview
• Content Indexing & Metadata
Generation for image and video
• Media Analysis Solution

Amazon Machine Learning Stack
Platforms
Application services
A m a z o n
R e k o g n i t i o n
A m a z o n
R e k o g n i t i o n
V i d e o
P o l l y T r a n s c r i b e T r a n s l a t e C o m p r e h e n dL e x
Amazon SageMaker Amazon Mechanical Turk
Frameworks KERAS
NVIDIA
Tesla V100 GPUs
(14x faster than P2)
P3
Machine Learning
AMIs
5,120 Tensor cores
128 GB of memory
1 Petaflop of compute
NVLink 2.0
Infrastructure
&

Automating Footage Tagging with Amazon Rekognition
• Built in 3 weeks
• Indexed against 99,000 people
• Index created in one day
• Saved ~9,000 hours a year in
manual curation costs
• Live video with frame sampling
Previously, only about half of all footage was indexed due to the immense
time requirements required by manual processes

Key Services For Today
Processing ML Orchestration
AWS Step Functions
AWS LambdaAmazon
Transcribe
Amazon
Rekognition
AWS Elemental
MediaConvert
AWS Elemental
MediaLive

AWS Elemental MediaConvert
• Professional grade video features and quality
• No software or hardware infrastructure to manage
• Automatically scales in response to variations in
incoming video volume
• Ability to manage capacity and control order in
which jobs are processed
• Pay for what you use, billed by the second of
content produced
AWS Elemental
MediaConvert

Amazon Rekognition
Object & scene
detection
Facial
analysis
Face
comparison
Face
search
Celebrity
detection
Image
moderation
Text
detection
"Amazon Rekognition allows us to scalably
identify and track actors across millions of
frames of content with much higher reliability
than any other solution we've used.”
- Jared Browarnik, Co-Founder & CTO, TheTake
“Amazon Rekognition enables us to quickly
and efficiently add value through various
automated metadata tagging processes, and
images and video segments are much easier to
find for our enterprise and our customers.”
- Shane Murphy, Solutions Engineer, Scrippsnetworks

Amazon Rekognition Video
Object, scene &
Activity detection
Face
search
Facial analysis Activity pathing
Unsafe content
detection
Celebrity
detection
Text in images

Amazon Transcribe
A fully managed and continuously trained automatic speech
recognition (ASR) service that takes in audio and automatically
generates accurate transcripts
Support for audio
in many formats
and low fidelity
§
Amazon S3
integration
Hello
/Hola
Time stamps and
confidence scores
English and
Spanish
Punctuation

Amazon Transcribe – Use Cases
Call Center
Subtitles
MeetingsContent indexingCompliance

AWS Step Functions
VisualizeDefine Monitor
Task A single unit of work
Choice Adds branching logic
Parallel Fork and join the data
Wait Delay for a specified time
Fail Stops an execution
Succeed Stops an execution successfully
Pass Passes its input to its output

Using Amazon ML Services for Media
• Use services such as Amazon
Rekognition & Amazon Transcribe to
generate metadata about your content
• Store that metadata and make it
searchable
• Retrieve only the portion of the
content you want
• Prepare it for timely use
Live and file
Sources
Amazon ML
Services
ML
Amazon
DynamoDB
Database
Live and file
Content
Content Indexing / Metadata Generation Content Retrieval / Action Metadata
AWS Elemental
Media Services
Media
processing
AWS Elemental
Media Services
Media
processing

Content Indexing / Metadata Generation

Content Indexing / Metadata Generation
File-
based
content
Live
content
MediaLive
Kinesis
Video
Streams
MediaConvert
Amazon Rekognition
(Image)
• JPEG/PNG
• Up to 15 MB
Amazon Rekognition
(Video)
• H.264 video
• MP4/MOV file
• Up to 8 GB
Transcribe
• FLAC/MP3/WAV/
MP4
• Up to 2 hours
• Up to 1 GB

Content Indexing / Metadata Generation − Image
AWS Elemental
MediaConvert job
transcodes file and
extracts JPEG frames
to S3 bucket.
AWS Lambda function
triggered by Amazon S3
object-created event tells
Amazon Rekognition to
analyze the JPEG file.
Amazon Rekognition
performs requested
operation on image (i.e.,
object detection, celebrity
recognition, etc.).
Amazon Rekognition returns
result to AWS Lambda, which
stores tags and confidence
scores in Amazon DynamoDB,
Amazon Redshift, Amazon
Elasticsearch Service, Amazon
RDS, or whichever service best
suits the use case.
AWS Elemental
MediaConvert
File-based
processing
Amazon S3
Storage
AWS Lambda
Serverless
Amazon
Rekognition
ML / AI
Amazon
DynamoDB
Database
File
Source

MediaConvert Configuration
Framerate
determines the
number of images
that will be
extracted from the
video per second.
1/5 indicates to
create one JPEG
every 5 seconds.

Lambda Function Code Example – Amazon Rekognition
Image

Amazon Rekognition Image Results

Content Indexing / Metadata Generation – Video
job transcodes source file to
H.264/MP4 at a bit rate such
that the file size is <8 GB.
AWS Lambda function triggered
by Amazon S3 object-created
event tells Amazon Rekognition
to analyze the video file.
Amazon Rekognition Video
performs requested operation
on video (i.e., person tracking,
celebrity recognition, etc.).
Amazon Rekognition returns result
to AWS Lambda, which stores tags
and confidence scores in Amazon
DynamoDB, Amazon Redshift,
Amazon Elasticsearch Service,
Amazon RDS, or whichever service
best suits the use case.
AWS Elemental
MediaConvert
File-based
processing
Amazon S3
Storage
AWS Lambda
Serverless
Amazon
Rekognition
ML / AI
Amazon
DynamoDB
Database
File
Source

Use Container
MPEG-4 Container
(MP4) and a file
extension of mp4.
Set Video Codec to
MPEG-4 AVC
(H.264).
Select bit rate
accordingly so output
file is smaller than 8 GB.
For example, a 60-
minute movie at 7 Mbps
will be approximately
3.2 GB.

Lambda Function Code Example – Amazon Rekognition
Video

Content Indexing / Metadata Generation –
Transcription
job transcodes source file,
creating audio-only rendition
for Amazon Transcribe
AWS Elemental
MediaConvert also
creates normal
audio/video output
AWS Lambda function
triggered by Amazon S3
object-created event creates
a new Transcribe job
Amazon Transcribe
outputs JSON file of
detected words and timing
Lambda function converts
Amazon Transcribe JSON into
subtitle format (such as WebVTT,
SRT, or TTML) and delivers to
Amazon S3 bucket with content
AWS Elemental
MediaConvert
File-based
processing
AWS Lambda
Serverless
Amazon
Transcribe
ML / AI
File
Source
Amazon S3
Storage
Amazon S3
STORAGE

Lambda Function Code Example − Amazon Transcribe

Polling Transcribe with AWS Step Functions

Amazon Transcribe Results

Amazon Transcribe to Captions

AWS Media Analysis Solution
https://aws.amazon.com/answers/media-entertainment/media-analysis-solution/
• Generate searchable metadata from
your media assets using Amazon
Rekognition, Amazon Transcribe,
Amazon Comprehend, and Amazon
Elasticsearch Service
• Deploy in minutes with a single click
using AWS CloudFormation
• Interact via API or demo web UI
• Orchestrated with Step Functions,
extensible and easily customizable

Thank you
cclawton@amazon.com

Paul Macey
Specialist Solution Architect, Big Data and Analytics
AWS Public Sector
June 2019
Multi Source, Multi Speed
Data Consumption & Analytics on AWS
Immersion day

Agenda
How did it come to this?
Organisational goldmines
Well architected data pipelines
Multisource, multispeed patterns
Wrap up

How did it come to this?

Security
Day 0
Data governance
& metadata
Data centralised
& scalable
SQL & BI
ready
Analytical &
Data Science
foundation
Repeatable &
Extensible
Accelerated Data Lake

Unintended data pipelines
Geospatial Girl
The Oracle
The Data Scientist
External request
Sales
Analyst
Operations
Security Guy
The Boss

csv
json
xls
Databases
Batch
FTPStreaming
IoT

Well architected data pipelines
Operational
Excellence Security Reliability
Performance
Efficiency
Cost
Optimization

Customer example
Health Organisation
In New Zealand
Challenge :
Needed to bring disparate datasets together
Too many external files
Current State:
SQL Server 2008
Large text files
ETL (SSIS) https://www.linkedin.com/pulse/how-we-built-data-lake-less-than-4-weeks-alex-poor/
Solution :
Uses the ADL solution as a base
Extended the solution to access FTP server data
Entirely serverless

Multisource, multispeed
data and analytics patterns

Streaming

Streaming Pattern
Smart City
Camera
camera_stream
datetime
sensorId
locationId
currentTemperature
battery
objectDetected
status
camera_stream
Create SQL Schema Write SQL to query stream
Amazon Kinesis
Data Firehose
Amazon Kinesis
Data Streams
Amazon Kinesis
Data Analytics
Amazon Kinesis
Data Analytics

API Pattern - Inbound
WeatherTraffic
Amazon
CloudWatch
AWS Lambda Amazon S3

API Pattern - Outbound
Camera, weather, and traffic Insights
AWS Lambda Amazon S3Amazon
API Gateway

IoT and IoT Analytics
Find Hidden Data Connections with Enrichment
AWS IoT
Analytics
Device
Registry
Weather
data
AWS IoT Core

SFTP Pattern
+ No end-user disruption
+ Fully managed servers
+ Simple to use
+ Pay as you use
+ Native cloud integrations
AWS SFTP
Amazon S3
AWS Transfer
for SFTP

Data Stores
Data stores
CDC
AWS Database
Migration Service
* Target only
Amazon
DynamoDB*
Amazon
Kinesis
Amazon S3
Amazon RDS

Data Stores

"We migrated hundreds of our clients from our in-house data-center to Amazon RDS
Oracle 12c using the AWS Data Migration Service (DMS). Due to this service, we
could live-replicate the databases between our data-center and RDS before the
migration. That kept the migration down-time to the very minimum. We are very
happy with DMS and are planning to use it for Oracle to MySQL migration next”.
”The SCT Assessment Report was the key enabler to allow us to understand the
scope of effort required to complete an Oracle to PostgreSQL migration. What
was originally thought to be a largely manual task that no one was particularly
excited about having to do became a very straight-forward quick and easy
process."
“We are in the process of migrating some databases to Amazon Aurora. The ease
by which we can do this using the AWS Database Migration Service has
simplified this process for us and enabled us to accelerate our migration
efforts. The ability to closely monitor the process, the detailed logging feature, and
the support we received from AWS have given us a great deal of confidence in a
successful migration.”
Who is saying what about DMS and SCT?

Shared
file systems
NFS TLS
AWS storage resources
AWS DataSync
agent
AWS DataSync
Amazon S3
Amazon Elastic
File System
Region
Batch Processing Pattern
Corporate data center
Transfer data at speeds up to 10 times faster than open-source tools

New services, new patterns

AWS Lake Formation
Insights &
visualisation
Amazon
QuickSight
ETL
AWS Glue
Ad-Hoc
Querying
Amazon
Athena
Amazon S3
Storage
AWS Glue
Catalog / Connectivity

Amazon Neptune: A fully managed graph database
Amazon S3

AWS has services and patterns to create
YOUR organisations data goldmine
csv
json
xls
Batch
FTPStreaming
IoT
Geospatial Girl
The Oracle
Operations
External
Requests
The Data Scientist
Analyst
Sales

Success with well architected data pipelines
Security GuyThe Boss
The Oracle

References
AWS Accelerated Data Lake (Git)
https://github.com/aws-samples/accelerated-data-lake
AWS Accelerated Data Lake Blog (part 1 & 2)
https://aws.amazon.com/blogs/publicsector/from-data-silos-to-data-domains-bringing-common-data-together
https://aws.amazon.com/blogs/publicsector/securing-your-data-by-knowing-your-data
Our data lake story: How Woot.com built a serverless data lake on AWS
https://aws.amazon.com/blogs/big-data/our-data-lake-story-how-woot-com-built-a-serverless-data-lake-on-aws
Kinesis Data Generator
https://awslabs.github.io/amazon-kinesis-data-generator/

References
IoT analytics
https://aws.amazon.com/iot-analytics/
AWS Database Migration
Service
https://aws.amazon.com/dms/
AWS Transfer for SFTP
https://aws.amazon.com/sftp/
AWS Datasync
https://aws.amazon.com/datasync/
Amazon Kinesis
https://aws.amazon.com/kinesis/
Amazon Kinesis Analytics
https://aws.amazon.com/kinesis/data-
analytics

Available today @ GitHub
https://github.com/aws-samples/accelerated-data-lake
Includes
Data lake pipeline (CloudFormation)
Instructions
Data configuration, security and metadata templates
Delivery
Professional services
AWS partners
Accelerated Data Lake

AWS Public Sector
June 2019
Incorporating Vision & Intelligence
with Machine Learning at the Edge
Immersion day

Agenda
Recap: Machine Learning & IoT / Edge
Architectural models:
Hybrid AI at the Edge
At the Edge: How to deploy and retrain
Cloud-based only AI
Customer example: ParKam
Tools to make things easier
Well Architected
Resources

ML can enhance every application
Autonomous
vehicles
Smart
agriculture
Predictive
maintenance
Robotics
Speech and sound
recognition
Video
security
Anomaly
detection
More

M L F R A M E W O R K S &
I N F R A S T R U C T U R E
A I S E R V I C E S
R E K O G N I T I O N
I M A G E
P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E XR E K O G N I T I O N
V I D E O
Vision Speech Language Chatbots
A M A Z O N
S A G E M A K E R
B U I L D T R A I N
F O R E C A S T
Forecasting
T E X T R A C T P E R S O N A L I Z E
Recommendations
D E P L O Y
Pre-built algorithms & notebooks
Data labeling (G R O U N D T R U T H )
One-click model training & tuning
Optimization (N E O )
One-click deployment & hosting
M L S E R V I C E S
F r a m e w o r k s I n t e r f a c e s I n f r a s t r u c t u r e
E C 2 P 3
& P 3 N
E C 2 C 5 F P G A s G R E E N G R A S S E L A S T I C
I N F E R E N C E
Reinforcement learningAlgorithms & models ( A W S M A R K E T P L A C E
F O R M A C H I N E L E A R N I N G )
Artificial Intelligence (AI) & Machine Learning (ML)

Amazon SageMaker
Fully managed
hosting with auto-
scaling
One-click
deployment
Pre-built
notebooks for
common
problems
Built-in, high
performance
algorithms
One-click
training
Hyperparameter
optimisation
B U I L D T R A I N D E P L O Y

IoT virtuous cycle
Intelligence
and outcomes

AWS IoT Events
IoT virtuous cycle
AWS IoT Analytics
AWS IoT SiteWise
Intelligence
and outcomes
AWS IoT Device
Management
AWS IoT
Things Graph
AWS IoT Core
AWS IoT
Device Defender
AWS IoT
Device SDK
Amazon FreeRTOS
AWS IoT
Device Tester
AWS IoT
Greengrass

Build conditional logic to evaluate
incoming telemetry data to detect
events in equipment or a process
Detect events from data
across thousands of sensors
and other sources
Trigger responses to
optimize operations
AWS IoT Events allow you to continuously monitor data from your equipment and fleets of
devices for changes in operation and to trigger the appropriate response when events occur
Data
services
!
AWS IoT Events - Generally Available!

AWS IoT Things Graph - Generally Available!
Models Run at the
edge
Monitor
Flows
Visual UI

Data and
State Sync Security
Over the
Air UpdatesConnectors
Local
Device Shadows
Code
Deployment
Lambda Functions AWS-grade
security
Easily Update
Greengrass Core
Machine
Learning
Inference
Local Execution
of ML Models
Local
Resource
Access
Lambdas Interact
With Peripherals
Easy integrations
with AWS
services, protocol
adaptors and
other SaaS
providers
Local
Messages
and Triggers
Local
Message Broker
Manage
Secrets at
the edge
AWS Secrets
Manager
functionality
at edge
AWS Greengrass
Extend AWS IoT to the Edge

Scene
Model
AWS IoT
AWS
Greengrass
Project
Stream
(optional)
Device
AWS Elastic Beanstalk
Amazon
Rekognition
Amazon DynamoDB

Existing DeepLens Face Detection Model

New Face Detection Lambda to upload to S3

Cloud-side Lambda adds items to DynamoDB tables
when faces detected.

Demo - Deeplens using local and cloud inference
Coffee_blog.mp4 (3:23)

Inference Training
Machine Learning at the Edge
Local
actions
Edge Cloud
AWS IoT Analytics
Amazon S3

Inference Training
Local
actions
Edge Cloud
Amazon
SageMaker

Inference Training
Local
actions
Edge Cloud
Amazon SageMaker
Optimize with Amazon
SageMaker/Neo

Inference Training
Local
actions
Edge Cloud
AWS IoT Greengrass
Deploy

Inference Training
Local
actions
Edge Cloud
AWS IoT Greengrass

USB
Camera
Amazon SageMaker
Nvidia Jetson Nano
AWS IoT
Greengrass
IoT
topics
IoT
topics
AWS Cloud
Model

Nvidia Jetson Nano
GPU
128-core Maxwell
CPU
Quad-core ARM A57 @ 1.43 GHz
Memory
4 GB 64-bit LPDDR4 25.6 GB/s

Amazon Sagemaker training
• Caltech-256 data set
• Pre-built ‘image classification’ algorithm

USB
Camera
Amazon SageMaker
Nvidia Jetson Nano
AWS IoT
Greengrass
IoT
topics
IoT
topics
AWS Cloud
Model
At the Edge: Deploy and test

Demo - Inference at Edge
Beverages_p2.mp4 (3:00)

S3 Object Tags and Metadata
Image1001.jpg
jpeg image data
S3 object tags
S3 object
data.csv
S3 metadata
Key Value
Classification Internal
PII False
Use Case Interaction Extracts
Team Analytics
Key Value
Policy facility_iinternal
MD5 ab3116cded134
Data Owner User Interaction Team
Data Source prod_int_extraction_dw

USB
Camera
Amazon SageMaker
Nvidia Jetson Nano
AWS IoT
Greengrass
IoT
topics
IoT
topics
AWS Cloud
Model
At the Edge: Retrain

Video / Demo - Retraining, redeploy & test
Demo

Retraining and redeployment improvements to try
• Using Amazon SageMaker Ground Truth.
• Experimenting with training parameters in the Amazon SageMaker
notebook to improve your model’s accuracy.
• Adding a buffering mechanism to store captured images when the Core
device is offline and upload to S3 when connectivity to the cloud is
restored. Eg. Remote locations or poor connectivity

AI entirely in the Cloud
IoT thing
camera
Amazon Kinesis Video
Streams
Rekognition video
https://aws.amazon.com/blogs/machine-learning/easily-
perform-facial-analysis-on-live-feeds-by-creating-a-
serverless-video-analytics-environment-with-amazon-
rekognition-video-and-amazon-kinesis-video-streams/

Straight to Cloud -
https://aws-quickstart.s3.amazonaws.com/quickstart-onica-
connected-camera/doc/aws-iot-camera-connector-on-the-
aws-cloud.pdf

To make it easier…

Greengrass Image Classification connectors
The Image Classification connectors provide a machine learning (ML)
inference service that runs on the AWS IoT Greengrass core.
Training jobs built using ‘image classification’ can be accessed directly
from Greengrass ML image classification connector (Jetson Nano
support pending)
User-defined Lambda functions use the AWS IoT Greengrass Machine
Learning SDK to submit inference requests to the local inference
service.

Runtimes and precompiled framework libraries

Sagemaker NEO - Deep Learning Runtime

Problem: Framework optimization is complicated

Amazon SageMaker Neo: Train once, run anywhere
Compiler Processor vendors can integrate
hardware-specific optimizations
Neo
Runtime Device makers can embed runtime into
edge devices and IoT

SageMaker Neo is part of the SageMaker ML workflow
Train once, run
anywhere &
Model
Optimization
Train and
tune models
Amazon SageMaker Neo

Well architected IoT
Operational
Excellence Security Reliability
Performance
Efficiency
Cost
Optimization

IoT Architecture Resources
http://iotatlas.net/
https://d1.awsstatic.com/whitepapers/architectur
e/AWS-IoT-Lens.pdf
https://d1.awsstatic.com/whitepapers/Designing_MQTT_Topics_for_AWS_IoT_Core.pdf
https://www.aws.training/LearningLibrary?&search=IoT&tab=view_all

Resources
https://aws.amazon.com/blogs/machine-learning/track-the-number-of-coffees-consumed-using-aws-deeplens/
http://bit.ly/greengrass-ml-workshop
https://aws.amazon.com/blogs/iot/machine-learning-at-the-edge-using-and-retraining-image-classification-models-with-
aws-iot-greengrass-part-1/
https://aws.amazon.com/blogs/iot/machine-learning-at-the-edge-using-and-retraining-image-classification-models-with-
aws-iot-greengrass-part-2/
https://docs.aws.amazon.com/greengrass/latest/developerguide/ml-dlc-console.html
https://github.com/aws-samples/aws-greengrass-samples/blob/master/iot-blog/image-classification-connector-and-
feedback/notebook/greengrass-image-classification-blog.ipynb
https://aws.amazon.com/blogs/iot/how-to-install-a-face-recognition-model-at-the-edge-with-aws-iot-greengrass/
nVidia Jetson Nano:
https://aws.amazon.com/blogs/machine-learning/use-pre-trained-models-with-apache-mxnet/
https://devtalk.nvidia.com/default/topic/1049293/jetson-nano/i-was-unable-to-compile-and-install-mxnet-on-the-jetson-
nano-is-there-an-official-installation-tutorial-/1

Resources (cont.)
Cloud-based only AI
https://aws.amazon.com/blogs/machine-learning/easily-perform-facial-analysis-on-live-feeds-by-creating-a-serverless-
video-analytics-environment-with-amazon-rekognition-video-and-amazon-kinesis-video-streams/
https://aws.amazon.com/blogs/machine-learning/video-analytics-in-the-cloud-and-at-the-edge-with-aws-deeplens-and-
kinesis-video-streams/
https://aws.amazon.com/about-aws/whats-new/2018/10/kinesis-video-streams-fragment-level-metadata-support/
https://aws.amazon.com/quickstart/architecture/camera-connector-onica/

AWS Immersion Day - Image Data Insights & Analytics Specialist Session - June 2019

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a AWS Immersion Day - Image Data Insights & Analytics Specialist Session - June 2019

Semelhante a AWS Immersion Day - Image Data Insights & Analytics Specialist Session - June 2019 (20)

Mais de Amazon Web Services

Mais de Amazon Web Services (20)

AWS Immersion Day - Image Data Insights & Analytics Specialist Session - June 2019