Securing Your Big Data on AWS

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Securing Your Big Data on AWS
Hannah Marlowe, PhD
Professional Services Consultant

What to expect from this session?
• Current data challenges
• Data Lake approach
• Encryption in transit and at rest
• Components of a Data Lake
• Security best practices
• Storage
• Metadata/Catalog
• Compute
• Customer examples

Data challenge today
Quickly ingest and store
any type of data, at any
scale, and at low cost
Have a single source of
truth and quickly search
and find the relevant
data
Easily query the data
through a unified set of
tools

For Data to Be a Differentiator, Customers Need
to Be Able to…
• Capture and store new non-relational
data at PB-EB scale in real time
• New type of analytics that go beyond
batch reporting to incorporate real-time,
predictive, voice, and image recognition
• Democratize access to data in a secure
and governed way
New types of data
New types of analytics
Dashboards Predictive Image
Recognition
VoiceReal-time

Traditionally, Analytics Used to Look Like This
OLTP ERP CRM LOB
Data Warehouse
Business Intelligence • Relational data
• TBs–PBs scale
• Schema defined prior to data load
• Operational reporting and ad hoc
• Large initial CAPEX + $10K–$50K/TB/Year

Data Lakes Extend the Traditional Approach
Data Warehouse
Business Intelligence
OLTP ERP CRM LOB
• Relational and non-relational data
• TBs–EBs scale
• Diverse analytical engines
• Low-cost storage & analytics
• Help locate, curate, and secure your data
• Provide democratized access to data
within your organization
Devices Web Sensors Social
Big Data processing,
real-time, Machine Learning
Data Lake

Building a Data Lake on AWS
Athena
Query Service
Batch GlueIoT Lambda SageMaker

The primary components of a Data Lake
Architectural Layers of a
Data Lake (without security)
Storage
Compute
Metadata/Catalog
• Object storage – S3/Glacier
• Block storage - EBS
• File storage - EFS
• Attached instance store
• EC2 instance
• Redshift clusters

• Automatically Index data
• Easy search with
tags/business domain
• Curate and assign
relevancy score
• Easily commission and
decommission data sets
• Capture data lineage
Storage
Compute
Metadata/Catalog

• Server-based compute
• More than just standalone
EC2, also includes EMR,
Redshift
• Serverless compute (Lambda,
Athena, API Gateway, etc.)
• Hybrid
• Redshift SpectrumStorage
Compute
Metadata/Catalog

Ubiquitous Encryption

Encrypt data in transit
Protect data flows
Point “A” Point “B” Data flow protection
Enterprise data sources Amazon S3 Encrypted with SSL/TLS; S3 requests signed with AWS Sigv4
Amazon S3 Amazon EMR Encrypted with SSL/TLS
Amazon S3 Amazon Redshift Encrypted with SSL/TLS
Amazon EMR Clients Encrypted with SSL/TLS; varies with Hadoop application client
Amazon Redshift Clients Supports SSL/TLS; Requires configuration
Apache Hadoop on Amazon EMR
• Hadoop RPC encryption
• HDFS Block data transfer encryption
• KMS over HTTPS is not enabled by default with Hadoop KMS
• May vary with EMR release (such as Tez and Spark in release 5.0.0+)

Options for data-at-rest encryption in AWS
Client-side encryption
• You encrypt your data before submitting to an AWS service
• You supply encryption keys OR use keys in AWS Key Management Service under your
control
• Tools: AWS Encryption SDK, S3 Encryption Client, EMRFS Client, DynamoDB Encryption
Client
Server-side encryption
• AWS encrypts data on your behalf after it is received by the service
• 47 services including Amazon S3, Amazon EBS, Amazon RDS, Amazon Redshift,
Amazon WorkSpaces, Amazon Kinesis Streams, AWS CloudTrail…
• Integrated with AWS Key Management Service so that you control the keys

Server-Side Encryption in AWS
Two-tiered key hierarchy using envelope encryption
• Unique data key encrypts customer data
• Customer Master Keys encrypt data keys
Benefits
• Limits risk of compromised data key
• Better performance for encrypting large data
• Easier to manage small number of master keys
than billions of data keys
• Centralized access and audit of key activity
Customer master
keys
Data key 1
S3 object EBS volume Amazon
Redshift cluster
Data key 2 Data key 3 Data key 4
Custom
application
KMS

Storage
Compute
Metadata/Catalog
Storage

• Scalable, highly available storage
• Designed for 11 9’s of durability
• Central Data Lake source of truth
• Store anything at any scale
• Query in place with Athena and Redshift
Spectrum
Amazon S3

Secure S3 – preemptive controls
• Create buckets based on business domains
• Assign bucket policies
– Restrict by VPC, SSL, IP filters, KMS keys
• Restrict using Tags
• Enable encryption
• Enable versioning
• MFA delete
• Enable backups – across accounts/regions

Manage permissions with tags
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": "arn:aws:s3:::EXAMPLE-BUCKET-NAME/*"
"Condition": {"StringEquals": {"S3:ResourceTag/HIPAA":"True"}}
}
]
}

Secure S3 data – detective controls
• Enable AWS config to detect s3 bucket level changes
– s3-blacklisted-actions-prohibited, s3-bucket-policy-not-more-permissive, s3-bucket-
logging-enabled, s3-bucket-public-read-prohibited, s3-bucket-public-write-prohibited,
s3-bucket-replication-enabled, s3-bucket-server-side-encryption-enabled, s3-bucket-
ssl-requests-only, s3-bucket-versioning-enabled
• S3 data access audit using Cloudtrail – Log to separate cloudwatch logs
– Kerberos enabled EMR clusters allows you to track AD user
• Use Amazon GuardDuty to detect unauthorized and unexpected activity
• Enable Amazon Macie to classify sensitive data

• Fast, scalable data warehouse
• Deploy a data warehouse in minutes
• Massively Parallel Query Execution
• Column-based Storage
• Redshift Spectrum extends data
warehouse to data lake
Amazon Redshift

Amazon Redshift Authentication
BI tools SQL clientsAnalytics tools
Redshift
ADFS
Corporate
Active Directory
IAM
Amazon Redshift
ODBC/JDBC
User groups Individual user
Single Sign-On
Identity providers

Amazon Redshift Data Protection
In-Transit
• Amazon Redshift API calls are made over HTTPS
• SSL certificate for each Amazon Redshift cluster & SSL
required for connectivity
• Set require_ssl=true
At-Rest
• Enable cluster encryption on your Amazon Redshift cluster
• Supports client-side encryption using customer managed key
• Supports server-side encryption using SSE-KMS & SSE-S3

Redshift Audit Logging
• For all Redshift API calls
• For all KMS API calls
• For all S3 API calls
• STL_DDLTEXT
• STL_QUERYTEXT
• STL_UTILITYTEXT
• STL_STATEMENTTEXT
• Connection Log
• User Log
• User Activity Log
Enable the
enable_user_activity_logging
database parameter
AWS CloudTrail Amazon Redshift Amazon S3

Storage
Compute
Metadata/Catalog
Metadata/Catalog

• Central metadata repository
• Apache Hive metastore compatible
• Integrates with Hive, Spark, Presto,
Athena and Redshift spectrum
• Crawlers discover and classify data into
central searchable catalog
AWS Glue Catalog

Glue data catalog – Resource policies
• Fine-grained access control to Catalog using IAM policies
• Restrict what they can view and query

• Create centralized Data catalog
• Create a Data curator role in your organization
• Using IAM policies to control Catalog access – similar to S3 bucket policies
• Enable metadata encryption using AWS KMS
• Use IAM “Deny” for “glue:Delete*” operations
Glue Catalog Best practices

Storage
Compute
Metadata/Catalog
Compute

• Hadoop, Spark, Presto, Hive, more
• Easy to use, fully managed
• Launch a cluster in minutes
• Baked in security features
• Pay by the hour and save with Spot
Amazon EMR

Master instance
group
Core instance group
HDFS HDFS HDFS
Task instance group
Amazon EMR
Amazon
S3

Amazon EMR network security
• Create Clusters in private VPC subnets
• S3 access via VPC API endpoint
• AWS API access via NAT gateway
• Locked down security groups
• AWS Direct Connect from on-prem
AWS Direct
Connect
Amazon
EMR cluster
Private VPC subnet
Public VPC subnet
VPC NAT
gateway
S3 VPC
endpoint
AWS APIs
Amazon
S3

EMR authentication
• Configure Kerberos for cluster authentication
• Perimeter security using Apache Knox
• Simplify authentication of various Hadoop services and UI's
• Mask service specific URL's/Ports by acting as a Proxy
• Enable SSL termination at the perimeter
• Ease management of published endpoints across multiple clusters
Knox
Gateway
MIT Kerberos
{REST}

Amazon EMRFS
• Sits between Amazon EMR and and
Amazon S3
• EMR clusters use EMRFS for
reading and writing files from
Amazon S3
• Provides consistent view and data
encryption
• Use different IAM roles for EMRFS
requests to Amazon S3
• These IAM roles can be cluster
users, groups, or the location of
EMRFS data in Amazon S3
EMRFS
Amazon
EMR
Amazon
S3
AD Users
Data
Scientist
Developer
Business
User
IAM Roles
Analyst
Role
Developer
Role
Business
User Role
EMRFS
Security
Configuration

EMR – Data Protection
Amazon
S3
Amazon
EMR
At-rest data encryption
• For cluster nodes (EC2 instance volumes)
• Open-source HDFS encryption
• LUKS encryption for local volumes
HDFS
(Block-transfer
and RPC)
Local volumes
(Instance
Store/EBS)
In-transit data encryption
• For distributed applications
• Open-source encryption
functionality
In-transit data encryption
• For EMRFS traffic between S3 and
cluster nodes (enabled automatically
• TLS encryption
At-rest data encryption
• For EMRFS on S3
• Server-side or client-side encryption
• SSE-S3, SSE-KMS, CSE-KMS, or CSE-Custom

EMR Audit Logging
• For all EMR API calls
• For all KMS API calls
• For all S3 API calls
• /JobFlowId/node/
• /JobFlowId/steps/N/
• /JobFlowId/containers
AWS
CloudTrail
Amazon
EMR

Data Lake Solutions in the Wild

FINRA oversees > 3,000
securities firms doing
business in the United States.
Challenge:
FINRA’s legacy system did not
scale well
• Up to 75 billion events per day
• Run complex surveillance queries
over 20+ PB of data
Solution:
• Migrated their big data appliance
to a S3 Data Lake and used EMR
for ingestion and processing
• Migrated to RDS and testing Aurora

FINRA uses S3 to Build Data Lake with EMR
• Required fast access
across trillions of trade
records (20PB+)
• Migrated from
on-premises system
• Use Apache HBase on
Amazon EMR to store
and serve this data
• Use EMR engines—
Spark, Presto, and Hive
to process data
• Lower costs by 60% over
on-premises system
Spark
on EMR
Presto
on EMR
Hive
on EMR
S3
Herd
Metastore
HBase
on EMR

Nasdaq operates financial
exchanges around the
world, and processes
large volumes of data.
Challenge:
Nasdaq wanted to make their large
historical data footprint available
to analyze as a single dataset.
Solution:
• Use Amazon Redshift for
interactive querying
• Use Amazon S3 as a Data Lake,
and Presto on EMR to process
historical data

Nasdaq Uses AWS to Build a Data Lake
• Migrate legacy
on-premises warehouse
to Amazon Redshift
• 4.8B rows inserted
per trading day
(orders, trades, quotes)
• Ingest data from multiple
sources, validates, and
stages in S3
• Redshift reads data out of
S3 for fast queries
• Presto on EMR and S3 used
for analysis of massive
historical data set
Data from all 7 exchanges
operated by Nasdaq
(orders, quotes, trade executions)
Flat
files
Operational
Databases
EMR
Redshift
S3
SQL
Clients

Zillow operates a portfolio of
the largest online real-estate,
and home-related brands.
Challenge:
Needed to analyze 100M homes
with fast, massively parallel
machine-learning jobs.
Solution:
• Use Spark on Amazon EMR, and
Amazon Kinesis power Zillow’s
site (personalization, advertising
optimization, and recommendations)

Zillow Uses AWS to Build Personalized Website
• Ingests data public data
(property records, recent
sales) into Kinesis
• Spark on EMR performs
ML to provide real-time
home estimates
• Store data into S3
Data Lake
• Use data to do
personalization,
advertising optimization,
and recommendations
for website
Zestimates home
valuation estimates
Personalization, advertising
optimization, and
recommendations for website
Public-property records,
home tax assessments,
sales transactions, images,
video, MLS-listing data, and
user-provided data
Kinesis
S3
Spark
on EMR
Public
Data

Asurion is a leader in
providing customer support
and protection for smartphones,
tablets, consumer electronics,
and appliances.
Challenge:
Had a wide variety of relational (from
OLTP databases), and non-relational
data (from telephony, voice-to-text,
claims data, and social) from >290M
customers. Wanted to do analytics.
Solution:
• Use S3 as a data lake to store all
data in a single location
• Use EMR to process raw data
• Use Redshift for fast BI

Asurion Uses AWS for Data Lakes and Analytics
• Collect data from a variety
of sources –OLTP, from
telephony, voice-to-text,
claims data, and social
• Real-time data is streamed
with Kinesis and Lambda
• Data lands in S3 Data Lake
and processed with EMR to
process raw data
• Redshift provides fast
analytics for BI tools
Data from
other
applications
Data from
OLTP
Application
Events &
Logging
Domain
applications
EMR
Data preparation: Process raw
data into meaningful content
Access
Virtualization
(AD and User
Profiles)
3rd party
BI Tools
Orchestration – Jobs, Plans, Workflows – Enterprise Scheduler – Information Lifecycle Management
Real-time data
Redshift
DynamoDB
S3
Kinesis
Data
Collection
Services
(ODBC/
JDBC, CDC,
Event
Streaming,
APIs)

Airbnb is a community
marketplace that allows
property owners and travelers
to connect with each other.
Challenge:
Grows data 3x every year with PBs
of data stored. Use Hadoop/HDFS,
but experienced bottlenecks in
performance and high costs.
Solution:
• Created a tiered storage system:
Land hot data in HDFS, and all
warm/cold data in S3 data lake
• S3 provides infinite storage at
lower costs

Airbnb Uses AWS for data lake and analytics
• Land hot data in HDFS
• Warm/cold data in S3
• Brings the best of both—
performance, scalability, cost
• Analyze data with Hive,
Presto, Spark, etc.
Hive on EMR
HDFS Cluster
S3
Spark on EMR
Presto on EMR

Amazon.com’s vision is to be
the earth’s most customer—
centric company; where people
can find anything they want
to buy online.
Challenge:
Load 500K+ transactions each day, and
serve 300K+ queries/extracts each day
from Amazon businsses (Amazon.com,
Amazon Prime, Amazon Music, Amazon
Alexa, Amazon Video, and Twitch).
Solution:
• Land data in S3 as a data lake
• Use Redshift as preferred SQL based
analysis by business users, and EMR
for machine learning

Amazon.com Uses AWS for Data Lakes & Analytics
• DynamoDB capturing all
Amazon.com transactions
• Everything from
DynamoDB, RDS
PostgreSQL and Kinesis fed
to a S3 data lake
• Glue used to catalog
the data
• Redshift used for all SQL-
based queries, and EMR for
all machine learning and big
data processing
• End-users use QuickSight
for visualizations
AWS Glue
Catalog
QuickSight
S3 Athena
EMR
DynamoDB
PostgreSQL
Kinesis
Redshift
Machine
Learning

Thank you!
marloweh@amazon.com

aws.amazon.com/activate
Everything and Anything Startups
Need to Get Started on AWS

Securing Your Big Data on AWS

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Securing Your Big Data on AWS

Semelhante a Securing Your Big Data on AWS (20)

Mais de Amazon Web Services

Mais de Amazon Web Services (20)

Securing Your Big Data on AWS