In this workshop you'll explore approaches for processing data using serverless architectures. You'll build processing infrastructure to enable operations personnel in Wild Rydes headquarters to monitor the health of the unicorn fleet. Each unicorn is equipped with a sensor that reports its location and vitals and you'll explore approaches for processing this data in batches and real-time. To build this infrastructure, you will use AWS Lambda, Amazon S3, Amazon Kinesis, Amazon DynamoDB, and Amazon Athena. You'll create functions in Lambda to process files and streams, use DynamoDB to persist unicorn vitals, create a serverless application to aggregate these data points using Kinesis Analytics, archive the raw data using Kinesis Firehose and Amazon S3, and you'll use Amazon Athena to run ad-hoc queries against the raw data.

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SSID: Guest
Password: Cube@11999
WildRydes: Serverless Data Processing Workshop
Adir Sharabi, AWS Solutions Architect
Dima Breydo, AWS Solutions Architect
David Feldstein , AWS Solutions Architect
Alon Gendler, AWS Solutions Architect

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scenario: Wild Rydes (www.wildrydes.com)

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Build processing infrastructure to enable operations personnel
in WildRydes headquarters to monitor the health of the unicorn fleet
Your task
Welcome to Wild Rydes
Inc., Employee #4!

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Services You Will Use Today

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is serverless?
No infrastructure provisioning,
no management
Automatic scaling
Pay for value Highly available and secure

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Lambda
Services (anything)
Changes in
data state
Requests to
endpoints
Changes in
resource state
Event source Function
Node.js
Python
Java
C#
Go
Ruby
Runtime API

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Anatomy of a Lambda function
Handler() function
Function to be executed
upon invocation
Event object
Data sent during
Lambda function
Invocation
Context object
Methods available to
interact with runtime
information (request ID,
log group, more)

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Serverless architectures with AWS Lambda
Lambda invoked
File put
into bucket
Amazon S3
Object
AWS Lambda
Message inserted
into to a queue
2. Function
removes
message from
queue
1. Lambda polls
queue and
invokes function
Amazon Simple Queue
Service
Message
Amazon
SQS
AWS Lambda
Lambda invoked
Request sent to
API Gateway
API Call
AWS Lambda
Amazon API Gateway

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Serverless architectures
1. Lambda polls
stream
Data published to a
stream
2. Amazon
Kinesis returns
stream data
Data
Amazon Kinesis
Data Streams
AWS Lambda
Request sent to ALB
Application Load
Balancer
Request
AWS Lambda
Scheduled time
occurs
Event (time-base)
AWS Lambda

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Add user sign-up, sign-in, and data synchronization to your apps
Amazon Cognito
Add user sign-up and
sign-in to your
mobile and web apps
Federate identities
and provide secure
access to AWS
resources
Store and sync across
devices

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Fast and flexible NoSQL database service for any scale
Amazon DynamoDB
Dead simple
• GetItem(primaryKey)
• PutItem(item)
Robust depth
• Fine-grained access control
• Streams/Triggers
• Cross-region replication
• Global Tables
• TTL
• Auto Scaling
• Atomic counters
• Backup and restore
• On Demand (NEW)
• Transaction (NEW)

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Durable, massively scalable object storage
• Designed for 99.999999999% durability and
99.99% availability
• Stores trillions of objects and regularly handles
millions of requests per second
• Effectively infinite storage without provisioning
capacity
Amazon Simple Storage Service (Amazon S3)

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Streaming with Amazon Kinesis
Easily collect, process, and analyze video and data streams in real time
Capture, process, and
store video streams
Load data streams
into data stores
Analyze data streams with
SQL or Java Flink (New)
Capture, process, and
store data streams
Amazon Kinesis
Data Streams
Amazon Kinesis
Data Firehose
Amazon Kinesis
Data Analytics
Amazon Kinesis
Video Streams

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kinesis Data Streams
• Serverless
• Elastically scalable
• Low cost, pay for what you use
• Managed encryption
• AWS and open source integrations
• Near real-time applications
• Multiple consumers

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Streams: Processing a data stream with Lambda
Data
producer
KinesisData
Streams
Amazon
SNS
Continuously
stream data
Lambda
service
Lambda
functionA
Lambda
functionB
Continuously polls for
new data,
1 poll per second
Automatically invokes
your function(s) when
data found

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Streams: How it works
Data
producer
KinesisData
Streams
AWSLambda
service
functionA
(instance1)
shard1
shard2
shard3
shard4
functionA
(instance2)
functionA
(instance3)
functionA
(instance4)
functionB
(instance1)
functionB
(instance2)
functionB
(instance3)
functionB
(instance4)
functionC
(instance1)
functionC
(instance2)
functionC
(instance3)
functionC
(instance4)

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kinesis Data Firehose
• Serverless
• Zero administration
• Seamless elasticity
• Pay for usage
• Direct-to-data store integration
• Serverless ETL using AWS Lambda

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kinesis Data Firehose: Filter, enrich, and convert
data while it is streaming
Data
producer KinesisData
Firehose
Amazon
Elasticsearch
Service
[Wed Oct 11 14:32:52 2017] [error] [client 127.0.0.1]
[Wed Oct 11 14:32:53 2017] [info] [client 127.0.0.1]
geo-IPservice
{
"date": "2017/10/11 14:32:52",
"status": "error",
"source": "127.0.0.1",
"city": "Boston",
"state": "MA"
}
{
"recordId": "1",
"result": "Ok",
"data": {
"date": "2017/10/11 14:32:52",
"status": "error",
"source": "127.0.0.1",
"city": "Boston",
"state": "MA"
}
},
{
"recordId": "2",
"result": "Dropped"
}

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kinesis Data Analytics
• Serverless
• Zero administration
• Pay for usage
• Sub 1-second processing latencies
• Filter, aggregate, and enrich data using SQL
• Programming skills not required

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kinesis Data Analytics: Pre-process data before
analysis
Data
producer
KinesisData
Analytics
KinesisData
Streams|Firehose
Pre-
processor
SQL
engine
…
• Asynchronously
invokes pre-
processing
Lambda function
KinesisDataStreams
AWSLambda
KinesisFirehose
• Tumbling windows:
Lambda invoked at the
end of the window
• Sliding windows or
continuous queries:
invoked ~1 per second

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Serverless
• Query service for querying data directly from S3
• Use standard ANSI SQL queries
• Support for multiple data formats include text, CSV,
TSV, JSON, Avro, ORC, Parquet
• Pay per query - on data scanned; $5 per TB
• Query using AWS console, ODBC, JDBC or QuickSight
• Improve performance and reduces cost:
• Compression, Partitioning & Columnar formats
Amazon Athena

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WildRydes Workshop
Overview

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• OBJECTIVE: Collect real-time data coming from the sensors attached to your
unicorn fleet. This sensors emits data every second including the unicorn's current
location, distance traveled in the previous second, and magic points and hit points
so that our operations team can monitor the health of our unicorns from Wild
Rydes headquarters
Module 1: Real-time Data Streaming

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• OBJECTIVE: Each minute, the application will emit data including the total distance
traveled in the last minute as well as the minimum and maximum readings from
health and magic points for each unicorn in our fleet
Module 2: Streaming Aggregation

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• OBJECTIVE: Process the aggregated stream and write those records to an Amazon
DynamoDB table.
Module 3: Stream Processing

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• OBJECTIVE: Every few minutes, Firehose will deliver the batched data to an
Amazon S3 bucket where it can be stored indefinitely for future reprocessing.
Using Amazon Athena, we'll run ad-hoc queries against our raw data.
Module 4: Data Lake

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Final Architecture

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Go to: https://dataprocessing.wildrydes.com/
• User accounts
• Your own accounts
• Be careful with production/your company/shared/etc. accounts
• Make sure all services are available in the region you are running in
• You can use N. Virginia, Oregon or Ireland
• We are here to help!
• Make sure to Cleanup your resources at the end of the Workshop.
Let the Fun Begin!
SSID: Guest
Password: Cube@11999

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank You!
Please give us your feedback!