How to go from zero to data lakes in days - ADB202 - New York AWS Summit

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How to go from zero to data
lakes in days
Mehul A. Shah
GM, AWS Glue and AWS Lake
Formation
Amazon Web Services
A D B 2 0 2
Srinivas Ravilisetty
IT Analytics Lead
Alcon

Agenda
Trends driving the revolution
What are data lakes?
What’s hard today?
AWS Lake Formation makes data lakes easy. Demo!

Decision making used to…
OLTP
ERP CRM
LOB
Enterprise Data Warehouse
Business Intelligence
… revolve around the Enterprise
Data Warehouse (in the 90s – 00s)

Data
every 5 years
There is more data than
people think
15
years
live for
Data platforms need to
1,000x
scale
>10x
grows
Data no longer fits
Data is more diverse

There are more people
accessing data
that want to analyze it in
different ways
And there are more rules
around data use
Data Scientists
Analysts
Business Users
Applications
machine learning SQL analytics
scientific
real-time,
streaming
Broader workloads

The cloud is a game changer
Amazon S3: ubiquitous storage allows
you to centralize datasets
Want a single locus of control
Many scalable analytics engines
available on-demand,
pay-as-you-go
Amazon S3
AWS Direct Connect
AWS Snowball
AWS Snowmobile
AWS Database Migration Service
AWS IoT Core
Amazon Kinesis Data Firehose
Amazon Kinesis Data Streams
Amazon Kinesis Video Streams
On-premises,
Batch
Real-time,
Streaming

Data lake: The new information hub
A centralized repository that enables
you to secure, discover, share, and analyze
structured and unstructured data at any scale

More data lakes & analytics on AWS than anywhere else

Building data lakes can still take months

Typical steps of building a data lake
Setup storage1
Move data2
Cleanse, prep, and
catalog data
3
Configure and enforce
security and compliance
policies
4
Make data available
for analytics
5
Ingestion & cleaning Security Analytics & ML

Data preparation accounts for ~80% of the work
Building training sets
Cleaning and organizing data
Collecting datasets
Mining data for patterns
Refining algorithms
Other

Sample of steps required Find sources
Create Amazon Simple Storage Service (Amazon S3) locations
Configure access policies
Map tables to Amazon S3 locations
ETL jobs to load and clean data
Create metadata access policies
Configure access from analytics services
Rinse and repeat for other:
Datasets, users, and end-services
And more:
manage and monitor ETL jobs
update metadata catalog as data changes
update policies across services as users and permissions change
manually maintain cleansing scripts
create audit processes for compliance
…
Manual | Error-prone | Time consuming

AWS Lake Formation: Build a secure data lake in days
Amazon S3
Data Lake Storage
Data
Catalog
Access
Control
Data import
Lake Formation
Crawlers ML-based
data prep
Simplifies data preparation:
ingest and cleaning
Secure data and efficiently
multiplex across engines
Self-serve: discover, share,
and collaborate

Demo: Build a data lake w/AWS CloudTrail data
Setup storage1
Move data2
Cleanse, prep, and
catalog data
3
policies
4
Make data available
for analytics
5
Ingestion & cleaning Security Analytics & ML

Demo Step 1: Register an Amazon S3 path as data lake
location

Demo Step 2 & 3: Load data w/ blueprint

Blueprints: Easily load data into your data lake
Logs
DBs
Prebuilt templates for different
data source types
Utilizes AWS Glue workflows
ETL jobs and crawlers automate
data layout, formats, and
partitions
Both one-time and incremental data loads
Populates common Data Catalog

Blueprints create AWS Glue workflows

Lake Formation is built on AWS Glue
Blueprints
Glue ETL Jobs
Workflow
Glue Crawlers
Glue Data Catalog
Connections,
Databases, Tables
Monitoring
Security, search, collaboration
AWS Glue
AWS Lake Formation

OrchestrationData Catalog Serverless Engine
Automatic crawling
Apache Hive Metastore compatible
Integrated with AWS analytic services
Discover
Flexible workflows
Monitoring and alerting
External integrations
Deploy
Apache Spark
Python shell
Interactive and batch jobs
Develop
AWS Glue components

Organize in Data Catalog: Databases and tables

Search and collaborate across multiple users
Text-based search across all
metadata
Add attributes like data
owners, stewards, and other
as table properties
Add data sensitivity level,
column definitions, and
others as column properties
Text-based
search and
filtering
Query data in
Amazon Athena

“FindMatches” ML Transform
Record matching
Findingtherelationshipsbetweenmultipledatasets,
evenwhenthosedatasetsdonotshareanidentifier
(orwhentheiridentifierisunreliable)
Deduplication
Transformingadatasetthathasmultiplerowsreferring
tothesameactualthingintoadatasetwherenotwo
rowsrefertothesameactualthing
ML FindMatches

Easily de-duplicate your data with ML transforms

Fuzzy de-duplication – Under the hood
State-of-the-art
Cost/accuracy
slider
Positive and negative
examples
Precision/recall
slider
Tune via:

Fuzzy de-duplication – Innovations
400M+
7.5B+
2.5

Ingestion & cleaning
Demo: Build a data lake w/AWS CloudTrail data
Setup storage1
Move data2
Cleanse, prep, and
catalog data
3
policies
4
Make data available
for analytics
5
Security Analytics & ML

Lake Formation: Centralized authorization
Data
Catalog
Access
Control
Lake Formation
Amazon S3
Data Lake Storage

Security permissions in Lake Formation
Control data access with simple grant
and revoke permissions
Specify permissions on tables and
columns rather than on buckets and
objects
Easily view permissions granted to a
particular user
Audit all data access in one place User 1
User 2

Security – Deep dive
User
IAM users, roles, and
Active Directory Amazon S3

Security – Take away
No intermediary in data path
End analytics services filter data
Lake Formation securely logs
all accesses for auditing

Demo Step 4: Grant permissions to users

Demo Step 5: Run query in Amazon Athena
Athena integration
guarantees user sees only the
tables & columns they have
access to
All access is logged and
auditable

Demo Step 5: Run query in Amazon Redshift
Redshift integration
guarantees user sees only
the tables & columns they
have access to
All access is logged and
auditable

Demo Step 6: Audit and monitor in real time
See detailed alerts in the console
Download audit logs for further
analytics
Data ingest and catalog
notifications also published to
Amazon CloudWatch events

Upgrading AWS Glue Data Catalog to Lake Formation
Lake Formation & AWS Glue use the same Data Catalog
Existing Glue crawlers, jobs, triggers and workflows will not change
Existing access to Glue resources will still be governed by IAM policies
Explicitly upgrade each data location, database & table when ready
Upgrading enables you to take advantage of new permissions model
Data
Catalog
ETL Jobs
Access
Control
Crawlers

AWS Lake Formation Pricing
No additional charges – Only pay for
the underlying services used

Conclusion
AWS Lake Formation makes setting up, securing, and
using data lakes simple
Data lakes are the evolution of warehousing
Lots more to come:
Simplify permissions: tag-based access control
provenance and compliance: lineage
ML-based controls: e.g., PII detection

Alcon
Srinivas Ravilisetty
IT Analytics Lead
Alcon

Alcon Analytics Team
Alcon Analytics Platforms
Tableau, Qlik, Alteryx, etc.
Alcon Data Lake Platform
Data Lake Platform on AWS
Alcon Data Management Platform
EBX5 Platform, Reference Data Management, Data Governance Teams, etc.
Analytics Products
Field Force Reporting, Sales Reporting, etc.
Advanced Analytics Products & Tools
R Studio, DataRobot, Forecasting, etc.

Problem statement & approach
Significant effort to get to trusted data
Limited data ownership & stewardship
Limited agility & extensibility
Global Source of trusted data for analytics
Business ownership and management of data
Rapid, continuous delivery of data and insights
Emerging and advanced analytic opportunities
Self-service analytics framework

Data lake journey
Data lakes on Hadoop (6)
Data warehouses (14)
Data marts (30+)
100+ servers
Multiple security models
Data Lake on Amazon S3
Storage and compute separated
Leverage AWS Services
Consistent and simplified
security model

Alcon data lake – Built using AWS
Simple user experience – Browse data assets, request access, perform
additional data preparation if needed, and analyze using service and
tool of choice to draw insights
Enables our users to analyze data more and prepare less
Separate storage and compute
• Storage using Amazon S3
• Compute using AWS Glue, Athena, Redshift, AWS Lambda, and
Amazon EMR
Built and released using Agile development methods on AWS

Third party &
Distributor
Files
CLOUD DATA
VISUALIZATION
INGEST STORE TRANSFORM PROVISION
Data
Privacy Tool
DATA MANAGEMENT
ANALYTICS
ForeSight - Alcon DataLake
ORA AnalyzOR
Data
Manufacturing
Data
On-premise FilesManaged Markets
Data
On-premise
Data Sources
ON-PREMISE
DATA
ERP
ANALYSE
INTEGRATION

Current security model – Complex & limiting
Current security management is complex due to security
enforced through S3 bucket polices
Complexity in providing access to data assets through
visualization and analytics tools
Limitation on size of policy files does not provide a scalable
solution
Column and row-level security achieved though complex policy
files or data replication, which is not sustainable

Why Lake Formation? Security
AWS Lake Formation simplifies our security model and data
cataloging
Reduces our effort in implementing:
• Reduced security model by 3x
• Reduced security management by 2x
• Eliminates custom code for security management
Reduces data redundancy

Why Lake Formation? Democratized Access
Expands our AWS Glue catalog metadata with business metadata
• Enables user-friendly search of data using
business metadata
Migration is simple and has minimal impact on users
• Existing tools work seamlessly

How to go from zero to data lakes in days - ADB202 - New York AWS Summit

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a How to go from zero to data lakes in days - ADB202 - New York AWS Summit

Semelhante a How to go from zero to data lakes in days - ADB202 - New York AWS Summit (20)

Mais de Amazon Web Services

Mais de Amazon Web Services (20)

How to go from zero to data lakes in days - ADB202 - New York AWS Summit