© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Markus Kaiser, Solutions Architect
Daniel Geske, Solutions Architect
Wild Rydes: Dawn of a new unicorn
Serverless data processing on AWS

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
What to expect
1. Serverless architectures
2. Wild Rydes scenario
3. Modules and relevant AWS services
• Setup
• Real-time data streaming
• Stream aggregation
• Stream processing
• Data lake
• Extra credit (optional)
4. Summary and clean-up

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Serverless
Build and run applications and
services without thinking of servers
Fully Managed
Developer Productivity
Continuous Scaling

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Help Wild Rydes Disrupt Transportation!
So how does this magic work?

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Wild Rydes is Backed by Leading Investors
THE BARN
ACCELERATOR
TENDERLOIN
CAPITAL
PENGLAI COMMUNICATIONS
AND POST NEW CENTURY
TECHNOLOGY CORP LIMITED

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Your Task: Process and visualize data in real-time
Welcome to Wild Rydes Inc.,
Employee #3!

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Serverless data processing on AWS
http://github.com/aws
-samples/aws-
serverless-workshops

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Setup
Configure your AWS Cloud9 IDE and setup pre-requisites like an AWS
Account.

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Real-time Streaming Data
Create a stream in Kinesis and write
to and read from the stream to
track Wild Rydes unicorns on a live
map. In this module you’ll also
create an Amazon Cognito identity
pool to grant the live map access to
your stream.
https://dataprocessing.wildrydes.com/streaming-data.html

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Amazon Kinesis Data Streams
• Easy administration & low cost
• Secure, durable storage
• Build real time applications
with framework of choice

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Amazon Kinesis Data Streams
Producers Consumers
Kinesis Agent
Apache Kafka
AWS SDK
LOG4J
Flume
Fluentd
AWS Mobile SDK
Kinesis Producer
Library
Get* APIs
Kinesis Client Library
+ Connector Library
Apache Storm
Amazon EMR
AWS Lambda
Apache Spark
Amazon
Kinesis

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Stream Aggregation
Build an Kinesis Data Analytics application to read from the
stream and aggregate metrics like unicorn health and distance
traveled each minute.
https://dataprocessing.wildrydes.com/streaming-aggregation.html

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Amazon Kinesis Analytics
• Connect to streaming source
• Write SQL code to process
streaming data
• Continuously deliver SQL results
• Fully managed, automatic elasticity

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Stream Processing
Persist aggregate data from application to a backend database
stored in DynamoDB and run queries against those data.
https://dataprocessing.wildrydes.com/stream-processing.html

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Data Lake
Use Kinesis Data Firehose to flush the raw sensor data to an S3
bucket for archival purposes. Using Athena, you’ll run SQL queries
against the raw data for ad-hoc analysis.
https://dataprocessing.wildrydes.com/data-lake.html

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Amazon Kinesis Data Firehose
• Zero administration and
seamless elasticity
• Direct-to-data store integration
• Serverless, continuous data
transformations

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Amazon Athena
No loading of data, analyze data directly from
Amazon S3
Standard SQL
• Support for complex data types (arrays, structs)
• Support for partitioning of data by any key
Query data in its raw format
• Athena supports multiple data formats:
Text, CSV, TSV, JSON, weblogs, AWS service logs
• or convert to an optimized form like ORC or Parquet
for the best performance and lowest cost

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Extra credit
AWS SDK
Amazon Kinesis Data Analytics
AWS Lambda
Amazon Athena

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Summary

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Serverless Data Processing Architecture

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Pricing
AWS Service Price* Example
Kinesis Firehose Data Volume: $0.033 per GB 1350 GB: $44.55
S3 Data Volume: $0.0245 per GB
PUT Requests: $0.0054 per 1.000
27 GB: $0.66
45.000 PUT: $0.243
Athena Scanned Data: $5 per TB 5.4 TB: $27.00
AWS Service Price* Example (Monthly)
Kinesis Streams Shard: $0.018 per hour
PUT Units: $0.0175 per Mio
2 x Shard: $27.00
270M PUTs: $4.725
Kinesis Analystics KPU: $0.132 per hour 1 x KPU: $99
Lambda Requests: $0.20 per Mio
Execution: $0,00001667 per GB/s (negligible)
1.000 Unicorns, 10s Status Msg, Msg Size 1KB
eu-central-1
30 days retention, 1:10 compression, 200 queries

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Clean up
1. Amazon Athena
2. Amazon Kinesis Data Analytics
3. Amazon Kinesis Data Firehose
4. Amazon Kinesis Data Streams
5. Amazon S3
6. AWS Lambda
7. Amazon DynamoDB
8. AWS IAM
9. Amazon Cognito
10.Cloud9 IDE

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Go build
Markus Kaiser, Solutions Architect
@markuskaiser
Daniel Geske, Solutions Architect
@btx94