In this webinar, discover how to develop accelerated data lake approaches to storing data with data security and governance from day 0. Join us as we discuss AWS services, architecture, and data onboarding methodologies that support these solutions. Presenter: Paul Macey, Big Data & Analytics, Specialist Solutions Architect, AWS

1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Paul Macey
Specialist Solution Architect, Big Data & Analytics
AWS Public Sector
Accelerated Data Lakes
Deep Dive Webinar

2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
Organisational data challenges
Accelerated data lake
Architecture
Onboarding
Demonstration
Wrap up
Questions

3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Organisational data challenges
Silos Governance
?
ScalabilitySecurity

4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Accelerated Data Lake
Security
Day 0
Data governance
& metadata
Data centralised
& scalable
SQL & BI
ready
Analytical &
Data Science
foundation
Repeatable &
extensible

5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Available today @ GitHub
https://github.com/aws-samples/accelerated-data-lake
Includes
Data lake pipeline (CloudFormation)
Instructions
Data configuration, security and metadata templates
Delivery
Professional services
AWS partners
Accelerated Data Lake

6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Accelerated Data Lake
Flywheel of success
Start Small
Establish a
Repeatable
Workflow
Deliver
benefits
Improve
and Iterate
Repeat

7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Architecture

8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Accelerated Data Lake
High Level Data Flow
Lambda Functions
• Validation
• Apply Security
• Attach Metadata
• Catalog object
• File movement
• Alerts
Time based or
Event Driven
ProcessInitiation
S3 buckets
• Staging
• Raw
• Curated
• Gold
• Data discovery
• Logs
Data Lake
Storage
Metadata
Data Catalog
Data Lake
Enabling Analytics and Insights
Big Data, Querying
ETL & ML
Database /
BI
Analytics & Insights

9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Accelerated Data Lake
File Processing Pipeline - S3 lambda event example
File arrives in the S3
Staging bucket
A lambda function is
triggered when the
object is created
The lambda passes the
S3 event data payload to
an AWS Step Function
The step function moves
through a repeatable data
file onboarding process
Validate Data Add security
tags to S3 object
Add metadata
to S3 object
Add object metadata
to DynamoDB
Index metadata
into ElasticSearch
Move file from the
Staging bucket to
Raw bucket
Get the
file specification

10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Onboarding

11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Accelerated Data Lake
Data onboarding process
1) Create an new file specification entry in a DynamoDb table
The table in the data lake solution is called “Data Sources”
1) Create a folder structure within the S3 staging bucket for the new data type
This is just to keep everything in S3 organised but also optimised for later use 

12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Specification Settings
File Settings S3 Object Tags
Simple Metadata
Extended Metadata

13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Accelerated Data Lake
Example S3 storage structure
Source Env /
Schema
Description /
Table / View
CMX
API-Raw
DB_Prod_HR
Staff
Data Flow Data FlowValidated & Approved
dev-rawdev-staging dev-curated dev-gold dev-validation
dev-data
discovery
sandpit
dev-logging
Wireless
CMX
2019
01
API-Raw
2019
01
select count(userid) from CMX where year=2019
select userid
from CMX
inner join API-RAW
on CMX.userid=API-RAW.userid
where year=2019

14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
S3 Object Tags and Metadata
Image1001.jpg
jpeg image data
S3 object tags
S3 object
data.csv
S3 metadata
Key Value
Classification Internal
PII False
Use Case Interaction Extracts
Team Analytics
Key Value
Policy facility_iinternal
MD5 ab3116cded134
Data Owner User Interaction Team
Data Source prod_int_extraction_dw

15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demonstration

16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
IoT Simulator
https://aws.amazon.com/solutions/iot-device-simulator/

17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demonstration

18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Wrap up
Security
Day 0
Data governance
& metadata
Data centralised
& scalable
SQL & BI
ready
Analytical &
Data Science
foundation
Repeatable &
Extensible
The accelerated data lake solution
Can enable your data
Support data security and data governance
Can grow and scale in harmony with your organisation
Can be granted access to AWS’s analytics, ML and AI ecosystem

19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
References
Amazon S3 security
https://aws.amazon.com/s3/faqs/#Security
https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html
AWS Accelerated Data Lake (Git)
https://github.com/aws-samples/accelerated-data-lake
AWS Accelerated Data Lake Blog (part 1 & 2)
https://aws.amazon.com/blogs/publicsector/from-data-silos-to-data-domains-bringing-common-data-together
https://aws.amazon.com/blogs/publicsector/securing-your-data-by-knowing-your-data
Our data lake story: How Woot.com built a serverless data lake on AWS
https://aws.amazon.com/blogs/big-data/our-data-lake-story-how-woot-com-built-a-serverless-data-lake-on-aws