1.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Paul Macey
Specialist Solution Architect, Big Data and Analytics
AWS Public Sector
November 2019
Multi Source, Multi Speed Analytics on AWS
Deep dive webinar

2.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
Sources of data
Organisational Goldmines
Well architected data pipelines
Combining multispeed data sources
Wrap up

3.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Geospatial
Operational Data Store Data ScientistBespoke datasets
Other data stores
Analyst
Operations
Sources of data
IoT

4.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Organisational goldmines
csv
json
xls
Databases
Batch
Streaming
IoT
SFTP

5.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Queensland Department of Transport and Main Road’s (TMR)
Connected and Autonomous Vehicle Initiative (CAVI)
https://www.itnews.com.au/news/queensland-debuts-most-advanced-driverless-car-in-oz-529501

6.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Analytics and Insights
ProcessInitiation Data Lake
Storage
Metadata /
Search
Big Data
Querying, ETL
& Insights
Database /
BI
Amazon
Athena
Amazon
QuickSight
AWS Glue
Amazon S3

7.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Combining multispeed data
sources

8.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
External Data Pattern
Weather
Amazon
EventBridge
AWS Lambda Amazon S3

9.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo
External data

10.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Streaming

11.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Generating streaming data
Kinesis Data Generator
https://awslabs.github.io/amazon-kinesis-data-generator/
For the streaming demonstration the generator will be producing
simulated smart city camera data

12.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Streaming Data Pattern
Smart City
Camera
camera_stream
datetime
sensorId
locationId
currentTemperature
battery
objectDetected
status
camera_stream
Create SQL Schema Write SQL to query stream
Amazon Kinesis
Data Firehose
Amazon Kinesis
Data Streams
Amazon Kinesis
Data Analytics
Amazon Kinesis
Data Analytics

13.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo
Streaming data

14.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SFTP Pattern – Reference Data
+ No end-user disruption
+ Fully managed servers
+ Simple to use
+ Pay as you use
+ Native cloud integrations
AWS SFTP
Amazon S3
AWS Transfer
for SFTP

15.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo
Reference data

16.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo
Combining streaming, external & reference
data

17.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
csv
json
xls
Databases
Batch
Streaming
IoT
AWS has services to bring together each of
YOUR organisational data goldmines
Geospatial
Operational Data Store
Operations
External
Requests
Data Scientist
Analyst
Sales
SFTP

18.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Available today @ GitHub
https://github.com/aws-samples/accelerated-data-lake
Includes
Data lake pipeline (CloudFormation)
Instructions
Data configuration, security and metadata templates
Delivery
Professional services
AWS partners
Accelerated Data Lake

19.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
References
AWS Accelerated Data Lake (Git)
https://github.com/aws-samples/accelerated-data-lake
AWS Accelerated Data Lake Blog (part 1 & 2)
https://aws.amazon.com/blogs/publicsector/from-data-silos-to-data-domains-bringing-common-data-together
https://aws.amazon.com/blogs/publicsector/securing-your-data-by-knowing-your-data
Our data lake story: How Woot.com built a serverless data lake on AWS
https://aws.amazon.com/blogs/big-data/our-data-lake-story-how-woot-com-built-a-serverless-data-lake-on-aws
Kinesis Data Generator
https://awslabs.github.io/amazon-kinesis-data-generator/

20.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
References
Amazon Kinesis
https://aws.amazon.com/kinesis/
Amazon Kinesis Analytics
https://aws.amazon.com/kinesis/data- analytics

21.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Webinar question resources – Data validation
https://json-schema.org/
https://frictionlessdata.io/docs/csv/
https://frictionlessdata.io/docs/csv/#libraries
https://pypi.org/project/CsvSchema/

22.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Webinar question resources - CDC
https://aws.amazon.com/blogs/database/load-cdc-data-from-relational-databases-to-amazon-kinesis-using-database-
migration-service/
https://aws.amazon.com/blogs/database/use-the-aws-database-migration-service-to-stream-change-data-to-amazon-
kinesis-data-streams/
https://aws.amazon.com/blogs/database/automating-database-migration-and-refreshing-activities-with-aws-dms/
https://aws.amazon.com/blogs/database/stream-changes-from-amazon-rds-for-postgresql-using-amazon-kinesis-data-
streams-and-aws-lambda/
https://docs.aws.amazon.com/en_pv/streams/latest/dev/kinesis-record-processor-ddb.html
https://aws.amazon.com/blogs/big-data/loading-ongoing-data-lake-changes-with-aws-dms-and-aws-glue/

23.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Webinar question resources – AWS Glue & JSON
https://aws.amazon.com/blogs/big-data/simplify-querying-nested-json-with-the-aws-glue-relationalize-transform/
https://docs.aws.amazon.com/en_pv/glue/latest/dg/add-classifier.html
https://docs.aws.amazon.com/en_pv/glue/latest/dg/aws-glue-programming-python-samples-legislators.html