Hadoop security implementationon 20171003

1. 1© Cloudera, Inc. All rights reserved.
Security Implementation on Hadoop
Dr. Wei-Chiu Chuang | Software
Engineer

2. 2© Cloudera, Inc. All rights reserved.
$ whoami
Software Engineer, Cloudera Apache Hadoop Committer/PMC

3. 3© Cloudera, Inc. All rights reserved.
Unguarded data stores are the victims

4. 4© Cloudera, Inc. All rights reserved.
Regulatory Compliance
Organizations can be fined up to 4% of
annual global turnover for breaching GDPR
or €20 Million

5. 6© Cloudera, Inc. All rights reserved.
Security Implementation

6. 7© Cloudera, Inc. All rights reserved.
Disclaimer
This talk serves as a general guideline for
security implementation on Hadoop.
The actual implementation procedures and
scope of implementation vary on a case-
by-case basis, and should be assessed by
Cloudera’s Professional Services team or
certified Cloudera SI Partners.

7. 8© Cloudera, Inc. All rights reserved.
Non-secure #0
Data Free for All

8. 9© Cloudera, Inc. All rights reserved.
Firewall
ActiveDirectory/KDC
Hadoop cluster
Cloudera
Manager
Gateway
node
Cloudera
NavigatorDatacenter
Applications

9. 10© Cloudera, Inc. All rights reserved.
High Availability made Easy

10. 11© Cloudera, Inc. All rights reserved.
Identity Management
Simple Authentication
File group ownership
• AD integration
• SSSD or Centrify
Consideration in large enterprises.
SSSD
via

11. 12© Cloudera, Inc. All rights reserved.
System Diagram #0
Firewall
ActiveDirectory
Master
Worker Worker Worker
Cloudera
Manager
Master
(SSSD/Centrify)

12. 13© Cloudera, Inc. All rights reserved.
Simple authentication =
no authentication

13. 14© Cloudera, Inc. All rights reserved.
Minimal Security #1
Reduce Risk Exposure

14. 15© Cloudera, Inc. All rights reserved.
Kerberos
EXAMPLE.COM
KDC
user@EXAMPLE.COM
Hadoop
user@EXAMPLE.COM 
user
Strong Authentication
KDC
• MIT
• ActiveDirectory (more common)
realmprimary

15. 16© Cloudera, Inc. All rights reserved.
Kerberos
Consideration in large corporates
Time synchronization
CM Kerberos Wizard
• Configure AD to create a Kerberos
principal for CM server, and to
delegate CM the ability to
create/manage Kerberos principals

16. 17© Cloudera, Inc. All rights reserved.
LDAPAuthentication
* LDAP over SSL

17. 18© Cloudera, Inc. All rights reserved.
Authorization/Access Control
HDFS File ACL YARN job submission
Hbase ACLsOozie ACL
Access Control List (ACLs)
Hive
Sentry Managed
(RBAC)
Impala

18. 19© Cloudera, Inc. All rights reserved.
Auditing

19. 20© Cloudera, Inc. All rights reserved.
Backup/Disaster Recovery
Cloudera Backup/Disaster Recovery (BDR)
• A high performance data replicator
• Copies incremental data on the source cluster at specified schedules
Supports
 Kerberos
 Data encryption
 HDFS replication to cloud

20. 21© Cloudera, Inc. All rights reserved.
Kerberized BDR Best Practice
Production DR
Cloudera BDR
PROD.EXAMPLE.COM
Cross-realm trust
KDC KDC
DR.EXAMPLE.COM

21. 22© Cloudera, Inc. All rights reserved.
Firewall
System Diagram #1
ActiveDirectory/
KDC
Master
Worker Worker Worker
Cloudera
Manager
Kerberos
Master
(SSSD/Centrify)
DR

22. 23© Cloudera, Inc. All rights reserved.
More Security #2
Managed, Secure, Protected

23. 24© Cloudera, Inc. All rights reserved.
Data In-Transit Encryption
RPC encryption
Data transport encryption
• Supports AES CTR, up to 256-bit
key length
HTTP TLS/SSL encryption
• No self-signed certificates in
production
Master
Worker Worker Worker
Master
Application
RPC encryption
Transport
encryption
TLS/SSL

24. 25© Cloudera, Inc. All rights reserved.
Data At-Rest Encryption
Transparent encryption
Supports any Hadoop applications
Encryption Zone
$ hadoop key create mykey
$ hadoop fs -mkdir /zone
$ hdfs crypto -createZone -keyName mykey -path /zone
/
/tmp
/zon
e
foo bar
Encryption zone

25. 26© Cloudera, Inc. All rights reserved.
Key Management Server Deployment (non-prod)
HDFS
NameNode
Client
Java
Keystore
KMS
Keystore
file
Separation of duties
• Encryption Zone Key (EZK) is stored in
KMS server
• HDFS super user can not decrypt files

26. 27© Cloudera, Inc. All rights reserved.
Key Management Server/Key Trustee Server Deployment
HDFS
NameNode
Client
Key Trustee
KMS
Key Trustee
KMS
Firewall
Key Trustee
Server
(Active)
Key Trustee
Server
(Passive)
synchronization
(or more)

27. 28© Cloudera, Inc. All rights reserved.
KMS+KTS+HSM Deployment
HDFS
NameNode
Client HSM KMS
HSM KMS
Firewall
Key Trustee
Server
(Active)
Key Trustee
Server
(Passive)
synchronization
Key HSM
(or more)
Key HSM
HSM
HSM

28. 29© Cloudera, Inc. All rights reserved.
Encryption Performance

29. 30© Cloudera, Inc. All rights reserved.
Troubleshooting: Encryption Performance Anomaly
• Configuration
• AES-NI Hardware acceleration
• OpenSSL library
• Entropy

30. 31© Cloudera, Inc. All rights reserved.
Fine Grained Access Control with Apache Sentry

31. 32© Cloudera, Inc. All rights reserved.
Firewall
System Diagram #2
ActiveDirectory/
KDC
Master
Worker Worker Worker
Cloudera
Manager
Kerberos
Master
KMSKMS
Firewall
KeyTrusteeKeyTrustee
(SSSD/Centrify)

32. 33© Cloudera, Inc. All rights reserved.
Most Security #3
Secure Data Vault

33. 34© Cloudera, Inc. All rights reserved.
Data Redaction
Personal Identifiable Information
• PCI-DSS, HIPAA
Best practice
Password
• stores in credential files, not in configuration
Log, queries
• Cloudera Manager

34. 35© Cloudera, Inc. All rights reserved.
Full Encryption
Encrypt Data Spills
• MapReduce
• Impala
• Hive
• Flume
OS-level encryption
• Navigator Encrypt

35. 36© Cloudera, Inc. All rights reserved.
Security Vulnerabilities

36. 37© Cloudera, Inc. All rights reserved.
Vulnerability Response and Process
Vulnerability
reports
Upstream
Internal
External
Fix Publish
CVE
Cloudera TSB

37. 38© Cloudera, Inc. All rights reserved.
Cloudera Certified Technology

38. 39© Cloudera, Inc. All rights reserved.
Cloudera Certified Technology Partners
Data Sources Data Ingest
Process, Refine
& Prep
Data Discovery Advanced Analytics
Connected
Machines/Data sources
Other Data Sources

39. 40© Cloudera, Inc. All rights reserved.
A certified product ensures it integrates with a secure
cluster
• Authenticate via Kerberos or LDAP
Authentication
• Handle Apache Sentry with Hive, Impala, Search, HDFS
Authorization
• Support HDFS transport encryption, at-rest encryption; support
SSL/TLS connection encryption
Encryption

40. 41© Cloudera, Inc. All rights reserved.
Cloudera SDX

41. 42© Cloudera, Inc. All rights reserved.
Cloudera Enterprise
42
The modern platform for machine learning and analytics optimized for the cloud
EXTENSIBLE
SERVICES
CORE SERVICES
DATA
ENGINEERING
OPERATIONAL
DATABASE
ANALYTIC
DATABASE
DATA CATALOG
INGEST &
REPLICATION
SECURITY GOVERNANCE
WORKLOAD
MANAGEMENT
DATA
SCIENCE
S3 ADLS HDFS KUDU
STORAGE
SERVICES

42. 43© Cloudera, Inc. All rights reserved.
• Unified security – protects sensitive data with consistent
controls, even for transient and recurring workloads
• Consistent governance – enables secure self-service access
to all relevant data and increases compliance
• Easy workload management – increases user productivity
and boosts job predictability
• Flexible ingest and replication – aggregates a single copy of
all data, provides disaster recovery, and eases migration
• Shared catalog – defines and preserves structure and
business context of data for new applications and partner
solutions
Open platform services
Built for multi-function analytics | Optimized for cloud

43. 44© Cloudera, Inc. All rights reserved.
Successful use cases

44. 45© Cloudera, Inc. All rights reserved.
Cloudera Overview & Financial Services Focus
2000
Strong Partner
Ecosystem
+
1600 Employees
Globally
+
19 Of the 30 G-SIBs Run
on Cloudera
Strong Focus &
Momentum in
Financial Services
3 Of the Fortune 500
Top 5 Insurers Run on
Cloudera
5 Of the Top 6 Asset
Management Firms
Run on Cloudera
200+
Financial Services
Customers

45. 47© Cloudera, Inc. All rights reserved.
Building a Fantastic Customer Experience
• Improved customer experience
• 80 percent reduction in operating costs
through a wide-range of customer
service and operational improvements
• Decrease in cost to service customers
while increasing revenue through better
service
CUSTOMER 360
FINANCIAL SERVICES
» PREDICTIVE ANALYTICS
» 360 CUSTOMER VIEW
» OPERATIONAL ANALYTICS

46. 48© Cloudera, Inc. All rights reserved.
Large healthcare
provider enables
practitioners to
recommend at-home
actions to prevent
hospital visits
• Flexible, automatic
data classification for
diverse medical
ontologies
• Self-service data
discovery for real-
time, data-driven
decisions

47. 49© Cloudera, Inc. All rights reserved.
Thank you
Wei-ChiuChuang | weichiu@cloudera.com

48. 50© Cloudera, Inc. All rights reserved.
More information on Hadoop Security

49. 51© Cloudera, Inc. All rights reserved.
Books authored by Clouderans