Today’s organisations require a data storage and analytics solution that offers more agility and flexibility than traditional data management systems. Data Lake is a new and increasingly popular way to store all of your data, structured and unstructured, in one, centralised repository. Since data can be stored as-is, there is no need to convert it to a predefined schema and you no longer need to know what questions you want to ask of your data beforehand. In this webinar, you will discover how AWS gives you fast access to flexible and low-cost IT resources, so you can rapidly scale and build your data lake that can power any kind of analytics such as data warehousing, clickstream analytics, fraud detection, recommendation engines, event-driven ETL, serverless computing, and internet-of-things processing regardless of volume, velocity and variety of data. Learning Objectives: • Discover how you can rapidly scale and build your data lake with AWS. • Explore the key pillars behind a successful data lake implementation. • Learn how to use the Amazon Simple Storage Service (S3) as the basis for your data lake. • Learn about the new AWS services recently launched, Amazon Athena and Amazon Redshift Spectrum, that help customers directly query that data lake.

1. Best Practices for
Building Your Data
Lake on AWS
Ian Robinson, Specialist SA, AWS
Kiran Tamana, EMEA Head of Solutions Architecture,
Datapipe
Derwin McGeary, Solutions Architect, Cloudwick

2. Amazon DynamoDB
Amazon Relational
Database Service
Amazon Redshift
p.39
Donotcreatetitlesthatarelarger
thannecessary.
Donotusetoosmallafont
sizeonmainorsubtext.
re
sizeof
Ffont.
Maintextgoeshere
70%-80%ofthefontsizeof
thetitle.WhitneyHTFfont.
Maintextgoeshere
70%-80%ofthefontsizeof
thetitle.WhitneyHTFfont.
subtextheresubtexthere
PTITLEOFSTEPTITLEOFSTEP
Maintextgoeshere70%-80%ofthe
fontsizeofthetitle.WhitneyHTFfont.
subtextherethatexplainsmovement
orprocessinbetweensteps
TITLEOFSTEP
Amazon S3 Athena
QueryService
Amazon Athena Amazon EMR
On-premise databases
Relational Database
NoSQL
Data Warehouse
Data Lake
Webinars: Database Freedom

3. Amazon DynamoDB
Amazon Relational
Database Service
Amazon Redshift
p.39
Donotcreatetitlesthatarelarger
thannecessary.
Donotusetoosmallafont
sizeonmainorsubtext.
re
sizeof
Ffont.
Maintextgoeshere
70%-80%ofthefontsizeof
thetitle.WhitneyHTFfont.
Maintextgoeshere
70%-80%ofthefontsizeof
thetitle.WhitneyHTFfont.
subtextheresubtexthere
PTITLEOFSTEPTITLEOFSTEP
Maintextgoeshere70%-80%ofthe
fontsizeofthetitle.WhitneyHTFfont.
subtextherethatexplainsmovement
orprocessinbetweensteps
TITLEOFSTEP
Amazon S3 Athena
QueryService
Amazon Athena Amazon EMR
On-premise databases
Relational Database
NoSQL
Data Warehouse
Data Lake
Webinar 1: Migrating Online/Transactional Workloads

4. Amazon DynamoDB
Amazon Relational
Database Service
Amazon Redshift
p.39
Donotcreatetitlesthatarelarger
thannecessary.
Donotusetoosmallafont
sizeonmainorsubtext.
re
sizeof
Ffont.
Maintextgoeshere
70%-80%ofthefontsizeof
thetitle.WhitneyHTFfont.
Maintextgoeshere
70%-80%ofthefontsizeof
thetitle.WhitneyHTFfont.
subtextheresubtexthere
PTITLEOFSTEPTITLEOFSTEP
Maintextgoeshere70%-80%ofthe
fontsizeofthetitle.WhitneyHTFfont.
subtextherethatexplainsmovement
orprocessinbetweensteps
TITLEOFSTEP
Amazon S3 Athena
QueryService
Amazon Athena Amazon EMR
On-premise databases
Relational Database
NoSQL
Data Warehouse
Data Lake
Webinar 2: Amazon Redshift and Data Warehousing

5. Amazon DynamoDB
Amazon Relational
Database Service
Amazon Redshift
p.39
Donotcreatetitlesthatarelarger
thannecessary.
Donotusetoosmallafont
sizeonmainorsubtext.
re
sizeof
Ffont.
Maintextgoeshere
70%-80%ofthefontsizeof
thetitle.WhitneyHTFfont.
Maintextgoeshere
70%-80%ofthefontsizeof
thetitle.WhitneyHTFfont.
subtextheresubtexthere
PTITLEOFSTEPTITLEOFSTEP
Maintextgoeshere70%-80%ofthe
fontsizeofthetitle.WhitneyHTFfont.
subtextherethatexplainsmovement
orprocessinbetweensteps
TITLEOFSTEP
Amazon S3 Athena
QueryService
Amazon Athena Amazon EMR
On-premise databases
Relational Database
NoSQL
Data Warehouse
Data Lake
Webinar 3: Data Lake Best Practices

6. Agenda
• What is a Data Lake?
• Key Data Lake Concepts
• Modern Data Architecture
• Putting it All Together
• Partner: Datapipe
• Partner: Cloudwick

7. What is a Data Lake?

8. What and Why
Store all your data, forever, at every stage of its lifecycle
Apply it using the right tool for the job
• Agile analytics
• Leverage all data that flows into your organization
• Customer centricity
• Better predictions via Machine Learning
• Competitive advantage

9. Data Lake versus Enterprise Data Warehouse
Complementary to EDW (not replacement) Data lake can be source for EDW
Schema on read (no predefined schemas) Schema on write (predefined schemas)
Structured/semi-structured/Unstructured data Structured data only
Fast ingestion of new data/content Time consuming to introduce new content
Data Science + Prediction/Advanced Analytics + BI use
cases
BI use cases only (no prediction/advanced analytics)
Data at low level of detail/granularity Data at summary/aggregated level of detail
Loosely defined SLAs Tight SLAs (production schedules)
Flexibility in tools (open source/tools for advanced
analytics)
Limited flexibility in tools (SQL only)
Enterprise DWEMR S3

10. Key Data Lake Concepts

11. STORAGE
COMPUTE
COMPUTE
COMPUTE
COMPUTE
COMPUTE
COMPUTE
COMPUTE
COMPUTE

12. Components of a Data Lake
Data Storage
• High durability
• Stores raw data from input sources
• Support for any type of data
• Low cost
Streaming
• Streaming ingest of feed data
• Provides the ability to consume any dataset as
a stream
• Facilitates low latency analytics
Storage & Streams
Catalogue & Search
Entitlements
API & UI

13. Components of a Data Lake
Catalogue
• Metadata lake
• Used for summary statistics and data
Classification management
Search
• Simplified access model for data discovery
Storage & Streams
Catalogue & Search
Entitlements
API & UI

14. Components of a Data Lake
Entitlements system
• Encryption
• Authentication
• Authorisation
• Chargeback
• Quotas
• Data masking
• Regional restrictions
Storage & Streams
Catalogue & Search
Entitlements
API & UI

15. Components of a Data Lake
Storage & Streams
Catalogue & Search
Entitlements
API & UI
API & User Interface
• Exposes the data lake to customers
• Programmatically query catalogue
• Expose search API
• Ensures that entitlements are respected

16. The Modern Data Architecture

17. data answers
COLLECT STORE
PROCESS/
ANALYZE
CONSUME
time to first answer

18. Agile Analytics
• Experiment
• Invest in promising experiments
• Fail fast
• React quickly

21. Storage
Ingest

22. Preservation of Original Source Data

23. Standard
Active data Archive dataInfrequently accessed data
Standard - Infrequent
Access
Amazon Glacier
Create
Delete
Events and Lifecycle Management

24. S3 as the Data Lake Fabric
• Unlimited number of objects and
volume
• 99.99% availability
• 99.999999999% durability
• Versioning
• Tiered storage via lifecycle
policies
• SSL, client/server-side encryption
at rest
• Low cost (just over $2700/month
for 100TB)
• Natively supported by big data
frameworks (Spark, Hive, Presto,
etc)
• Decouples storage and compute
• Run transient compute
clusters (with Amazon EC2
Spot Instances)
• Multiple, heterogeneous
clusters can use same data

25. Database Migration
Service
Automated Data Ingestion

26. Scalable (secure, versioned, durable) storage +
Immutable data at every stage of its lifecycle +
Versioned schema and metadata
=
Data discovery, lineage
Storage + Catalog

27. AWS Glue
• Data Catalog Discover and
store metadata
• Job Authoring Auto-
generated ETL code
• Job Execution Serverless
scheduling and execution

28. Hive metastore-compatible, highly-available
metadata repository:
• Search metadata for data discovery
• Connection info – JDBC URLs, credentials
• Classification for identifying and parsing
files
• Versioning of table metadata as schemas
evolve and other metadata are updated
• Table definitions – usable by Redshift,
Athena, Glue, EMR
Populate using Hive DDL, bulk import, or
automatically through crawlers.
Glue Data Catalog

29. AWS Lambda
AWS Lambda
Metadata Index
(Amazon DynamoDB)
Search Index
(Amazon Elasticsearch)
ObjectCreated
ObjectDeleted PutItem
Update Stream
Update Index
Extract Search Fields
Indexing and Searching Using Metadata
Amazon S3

30. Governance
Security
Privacy

31. Identity and Access Management
• Manage users, groups, and roles
• Identity federation with Open ID
• Temporary credentials with Amazon Security
Token Service (Amazon STS)
• Stored policy templates
• Powerful policy language
• Amazon S3 bucket policies

32. IAM
Amazon
S3
Amazon
ElastiCache
Amazon
DynamoDB
Amazon EMR Amazon
Kinesis
Amazon
Athena
Service API Access
Security at the Data Level

33. Third Party Ecosystem Security Tools
Amazon
S3
AWS
CloudTrail
http://amzn.to/2tSimHj
Amazon
Athena
Access Logging
API Logging
Access Log
Analytics
IAM
Amazon EMR
http://amzn.to/2si6RqS
Storage Level Support for Access Logging and Audit

34. Encryption Options
AWS Server-Side encryption
• AWS managed key infrastructure
AWS Key Management Service
• Automated key rotation & auditing
• Integration with other AWS services
AWS CloudHSM
• Dedicated Tenancy SafeNet Luna SA HSM Device
• Common Criteria EAL4+, NIST FIPS 140-2

35. Discovery
Search
Access

36. Data Lake API and UI
• Exposes the metadata API, search, and Amazon S3
storage services to customers
• Can be based on TVM/STS Temporary Access for
many services, and a bespoke API for metadata
• Drive all UI operations from API?

37. Amazon API Gateway
• Host multiple versions and stages of APIs
• Create and distribute API keys to developers
• Leverage AWS Sigv4 to authorize access to APIs
• Throttle and monitor requests to protect the backend
• Leverages AWS Lambda

38. Data Quality
Data Preparation
Orchestration and Job Scheduling

39. Job Authoring with AWS Glue
• Python code generated by
AWS Glue
• Connect a notebook or IDE
to AWS Glue
• Existing code brought into
AWS Glue

40. Job Execution with AWS Glue
• Schedule-based
• Event-based
• On demand

41. Instantly Query Your Data Lake on S3

42. AWS Batch

43. Write Database Changes to S3 with DMS
<schema_name>/<table_name>/LOAD001.csv
<schema_name>/<table_name>/LOAD002.csv
<schema_name>/<table_name>/<time-stamp>.csv
Full Load
Change Data Capture

44. Putting it All Together

45. Work backwards from product:
• Define application and analytics
datasets
• Identify and land raw sources of
data
• Cleanse, enrich, standardize to
create trusted sources
• Transform to create product
Raw Trusted Product
Organizing and Managing Your Data Lake

46. Organizing and Managing Your Data Lake
Structuring and Cataloguing
S3 prefixes – Separate and partition
data
AWS Glue Data Catalog – Versioned
metadata grouped by database/table
Amazon DynamoDB/Amazon
Elasticsearch – Indexing
Archiving
S3 lifecycle policies
Ownership, Classification,
Access Control
Buckets
Tags
Bucket policies
ACLs
Auditing
AWS CloudTrail and S3 Access
Logging

48. Building a Data Strategy on AWS
Kinesis Firehose
1
2
3
4
5
6
Athena
Query Service
7
8
Glue

49. Analytics Capabilities

50. Processing & Analytics
Transactional &
RDBMS
DynamoDB
NoSQL DB Relational Database
Aurora
BI & Data Visualization
Kinesis Streams
& Firehose
Batch
EMR
Hadoop, Spark,
Presto
Redshift
Data Warehouse
Athena
Query Service
AWS Batch
Predictive
Real-time
AWS Lambda
Apache Storm on
EMR
Apache Flink on
EMR
Spark Streaming on
EMR
Elasticsearch
Service
Kinesis Analytics, Kinesis
Streams
EastiCache DAX

51. https://aws.amazon.com/big-data/partner-solutions/
AWS Partner Ecosystem
Reduce the effort to move,
cleanse, synchronize,
manage, and automate data
related processes

52. DATAPIPE.COM US: 877 773 3306 UK: +44 800 634 3414 HK: +852 3521 0215
AWS DATA LAKES
Kiran Tamana, EMEA Head of Solutions Architecture, Datapipe

53. PREMIER CONSULTING PARTNER IN EMEA AND NORTH AMERICA
RECOGNISED AS A LEADER BY GARTNER WORLDWIDE
170+
YEARS
2.2k+
DAYS
10x
WINNER
9+
MILLION
4
CONTINENTS
Audited AWS Premier Consulting
Partner and AWS Direct Connect Partner
Proprietary Keyless Management
Global Presence and support Datapipe has Data Center locations in
29 datacenters across 9 countries
Stevie Award for
Customer Service
Monthly Instance
Hours Managed
Combined
AWS Professional
Services Experience
Experience Managing
AWS Since 2010

54. • Vessel tracking via satellite and terrestrial ID data
• Vessel ownership & characteristic data
• Vessel registration, safety and insurance data
• Casualty, arrest and vessel inspection data
• Fleet development data
• Global Ports and Terminal data
DATA LAKES – A MARITIME USE CASE
120,000
Tracking vessels
163,500
Shipping companies
2,800
Ports
19,000
Terminals
A requirement to manage a global network of multiple data types, coming
from a wide variety of business critical maritime sources

55. DEVELOP A MIGRATION STRATEGY
Review ongoing technical deployments
Identify existing pain points
Conduct a situational analysis of current system
Evaluate infrastructure for automation
Assess configuration usage
Recommend details on application deployment management

56. DATA LAKE ARCHITECTURE
CLIENT METHODOLOGY:
INGEST
STORE
DELIVER

57. Best Practices for Building Your Data Lake on AWS
Presented by
Derwin McGeary
Solutions Architect, Cloudwick EMEA
Contact us: services@cloudwick.com

58. Webinar Agenda
➢ About Cloudwick
➢ Reference Data Lake Architecture on AWS
➢ AWS Data Lake Implementations
➢ How to Get Started
Contact us: services@cloudwick.com

59. About Cloudwick
7
Founded
in 2010
3
Global Offices
US | EMEA | APAC
150+
Professionals
US | EMEA | APAC
Global
1000
Proven Success
400+
Certifications
Big Data, Analytics, Cybersecurity
Visualization & Cloud Partnerships
100+
Data Lake
Engagements
Contact us: services@cloudwick.com

60. Reference Data Lake Architecture on AWS
Marketing
Apps &
Databases
CRM
Databases
Other Semi-
structured data
Sources
Any Other
Data
Sources
Back office
Processing Data
Sources
Transformations/ETL
Curated
Layer
Raw
layer
Data Lake Data Lake
GLUE using PySpark/EMR
Data Ingestion Layer
Based on the data
velocity, volume and
veracity, pick the
appropriate ingest
services/other services
like Kafka, Flume, Sqoop,
DMS, Kinesis etc.
Data store Layer
This is the RAW layer for
the Data Lake. The
objects in the RAW layer
are immutable. The
structure to RAW layer is
typically partitioned by a
date/time
Curated Layer
This is the curated layer
for the Data Lake. The
data is ready for end
user. The
transformations/ETL
make the data useful.
Analytics Layer
Where the data is
modelled in the DWH.
This data is exposed
using JDBC/ODBC
connectors to the BI
tools, allowing for more
reuse and sharing.
Data Warehouse
Analytics
Layer
Visualizations
Business
Intelligence
S3 Storage
Access
Controlled
Zone
Secure and Controlled
access for Internal
Business Units and
3rd Party companies
API
Gateway
JDBC/ODBC
Native and 3rd Party BI and
visualisation tools
Access Layer
This layer allows for
innovative reuse, access,
visualisation, and
reporting on the data.
Contact us: services@cloudwick.com

61. AWS Data Lake Implementations
Contact us: services@cloudwick.com

62. SGN Use Case : Background & Challenges
62
➢ One of the Largest Gas Distribution Company
➢ Reporting Responsibilities
➢ Data Silos
➢ Challenges
Growing Data Sources
Existing ETL process
Too many silos
Data Provision for different use cases
Operational and Analytical Reporting
Contact us: services@cloudwick.com

63. Use Case: Data Lake and Migration for SGN
63
Contact us: services@cloudwick.com

64. Where to start?
Data Inventory - what data/metadata do you
already have?
Gap Analysis - what business and technical
metadata do you need for your use cases?
Value: Knowing the value of data and getting value
from data
Cloudwick can help you get started:
Contact us: services@cloudwick.com
Cloudwick Quickstart
http://tinyurl.com/CloudwickQuickStart
Cloudwick Jumpstart
http://tinyurl.com/CloudwickJumpStart

65. Thank you!