SlideShare uma empresa Scribd logo
1 de 40
Baixar para ler offline
© Hortonworks Inc. 2011
Improvements in Hadoop Security
Sanjay Radia
sanjay@hortonworks.com
@srr
Chris Nauroth
cnauroth@hortonworks.com
@cnauroth
Page 1
© Hortonworks Inc. 2011
Hello
Sanjay Radia
• Founder, Hortonworks
• Part of the Hadoop team at Yahoo! since 2007
– Chief Architect of Hadoop Core at Yahoo!
– Long time Apache Hadoop PMC and Committer
– Designed and developed several key Hadoop features
• Prior
– Data center automation, virtualization, Java, HA, OSs, File Systems (Startup, Sun Microsystems, …)
– Ph.D., University of Waterloo
Chris Nauroth
• Member of Technical Staff, Hortonworks
– Apache Hadoop Committer
– Major contributor to HDFS ACLs
• Hadoop user since 2010
– Prior employment experience deploying, maintaining and using Hadoop clusters
Page 2
Architecting the Future of Big Data
© Hortonworks Inc. 2011
Overview
• Models of Deployment
– Secure and insecure
• Hadoop Authentication
– The how and why
– Knox – perimeter security
• Authorization – existing and what is new
– HDFS
– Tables and Hive
– HBase and Accumulo
• Data protection and encryption
– Wire
– Data at rest
Page 3
Architecting the Future of Big Data
© Hortonworks Inc. 2011
Two Reasons for Security in Hadoop
Hadoop Contains Sensitive Data
– As Hadoop adoption grows so too has the types of data organizations look to store. Often the
data is proprietary or personal and it must be protected.
– In this context, Hadoop is governed by the same security requirements as any data center
platform.
Hadoop is subject to Compliance adherence
– Organizations are often subject to comply with regulations such as HIPPA, PCI DSS, FISAM that
require protection of personal information.
– Adherence to other Corporate security policies.
1
2
© Hortonworks Inc. 2011
Three Models of Hadoop Deployment
• Insecure cluster
– You have protected via the perimeter
– You trust the code that runs in the system
– - note In Hadoop cluster, user submitted code runs inside the cluster
– (Note true in typical client-server applications)
– The client side libraries pass the client’s login credential
– There is not end-end-authentication here
– Authorization is done against this credentials
• Secure cluster
– Full authentication
– Can run arbitrary code in jobs
• Perimeter security using Knox
– Internal cluster can be secure or insecure depending on your needs
© Hortonworks Inc. 2011
Pillars of Hadoop Security
Authorization
Restrict access to
explicit data
Audit
Understand who did
what
Data Protection
Encrypt data at rest &
motion
AD/Kerberos in native
Apache Hadoop
Perimeter Security
with Apache Knox
Gateway
Authentication
Who am I/prove it?
Control access to cluster.
© Hortonworks Inc. 2011
Hadoop Authentication Overview
• Kerberos/Active Directory based security
– SSO – users do not have to re-login into Hadoop
– Hadoop accounts do not have to be recreated
– Caveat – MR Task isolation require Unix accounts for each user current but this is going away with Linux containers
– Hadoop tokens – supplement the Kerberos authentication
– Delegation tokens – deal with the delayed job execution
– Block tokens – capabilities to deal with the distributed nature of HDFS
– Trusted Proxies – support for third party services to act as proxy
– Oozie
– Gateways – HDFS proxy, Knox, etc
• Base security infrastructure being extended to support other authentication services besides
Kerberos
• Knox – Perimeter Security and Rest Gateway
Page 7
© Hortonworks Inc. 2011
User DB/Account
Single-signon using your organization’s existing user DB
• No need to create Hadoop user accounts to be created
– Note the OS account on compute nodes are for isolation – this will be removed when we use Linux containers
• Currently supports Kerberos (and LDAP indirectly)
• The RPC layer’s authentication is fairly general
– Supports Kerberos and Hadoop’s tokens
– Can be extended
© Hortonworks Inc. 2011
Why the tokens
• Why does Hadoop have its own tokens?
– Standard client-server security model is not sufficient for Hadoop
– Works when logged-in client is directly accessing a Hadoop service
– But for a job, the execution happens much later
– The job submitter has long logged off
• Hence we needed to add delegation tokens
• HDFS is a distributed service and needed to add capability-like tokens for datanode
authentication
The permissions/ ACLs are in the Namenode
© Hortonworks Inc. 2011
Apache Knox
Perimeter Security for Hadoop REST APIs
Architecting the Future of Big Data
Page 10
© Hortonworks Inc. 2011
The Gateway or Edge Node
• Hadoop APIs can be used from any desktop after SSO login
– FileSystem and MapReduce Java APIs
– Pig, Hive and Oozie clients (that wrap the Java APIs)
• However it is typical to Use “Edge Node” or “Gateway Node” that is “inside” cluster
– The libraries for the APIs are generally only installed on the gateway
– Users SSH to Edge Node and execute API commands from shell
Page 11
HadoopUser Edge Node
SSH
© Hortonworks Inc. 2011
• Single Hadoop access point
• REST API hierarchy
• Consolidated API calls
• Multi-cluster support
• Eliminates SSH “edge
node”
• Central API management
• Central audit control
• Simple Service level
Authorization
• SSO Integration –
Siteminder, API Key*,
OAuth* & SAML*
• LDAP & AD integration
Perimeter Security with Apache Knox
Integrated with existing
systems to simplify identity
maintenance
Incubated and led by Hortonworks,
Apache Knox provides a simple and open framework for
Hadoop perimeter security.
Single, simple point of
access for a cluster
Central controls ensure
consistency across one or
more clusters
© Hortonworks Inc. 2011
Hadoop REST APIs
• Useful for connecting to Hadoop from the outside the cluster
• When more client language flexibility is required
– i.e. Java binding not an option
• Challenges (Knox addresses these challenges)
– Client must have knowledge of cluster topology
– Required to open ports (and in some cases, on every host) outside the cluster
Page 13
Service API
WebHDFS Supports HDFS user operations including reading files, writing to files,
making directories, changing permissions and renaming. Learn more about
WebHDFS.
WebHCat Job control for MapReduce, Pig and Hive jobs, and HCatalog DDL
commands. Learn more about WebHCat.
Hive Hive REST API operations
HBase HBase REST API operations
Oozie Job submission and management, and Oozie administration. Learn more
about Oozie.
© Hortonworks Inc. 2011
What can be done today?
Authorization
Restrict access to
explicit data
Audit
Understand who did
what
Data Protection
Encrypt data at rest &
motion
Previously
• All Services: Service level ACLs
• HDFS: Permissions
• Yarn: Queue ACLs
• Hive/Pig Tables: Table level via
HDFS
• Apache Accumulo: Cell level
• HBase: Namespace, Table,
Column Family and Column level
ACLs
Authentication
Who am I/prove it?
Control access to cluster.
Hadoop 2.x
• HDFS: ACLs
• Hive: Column level ACLs
• HBase: Cell level ACLs
• Knox:
• Rest Service level Authorization
• Access Audit with Knox
© Hortonworks Inc. 2011
HDFS ACLs
• Existing HDFS POSIX permissions good, but not flexible enough
– Permission requirements may differ from the natural organizational hierarchy of users and groups.
• HDFS ACLs augment the existing HDFS POSIX permissions model by implementing the POSIX
ACL model.
– An ACL (Access Control List) provides a way to set different permissions for specific named users or named
groups, not only the file’s owner and file’s group.
Page 15
Architecting the Future of Big Data
© Hortonworks Inc. 2011
HDFS File Permissions Example
• Authorization requirements:
–In a sales department, they would like a single user Maya (Department Manager) to
control all modifications to sales data
–Other members of sales department need to view the data, but can’t modify it.
–Everyone else in the company must not be allowed to view the data.
• Can be implemented via the following:
Read/Write perm for user
maya
User
Group
Read perm for group sales
File with sales data
© Hortonworks Inc. 2011
HDFS ACLs
• Problem
–No longer feasible for Maya to control all modifications to the file
– New Requirement: Maya, Diane and Clark are allowed to make modifications
– New Requirement: New group called executives should be able to read the sales data
–Current permissions model only allows permissions at 1 group and 1 user
• Solution: HDFS ACLs
–Now assign different permissions to different users and groups
Owner
Group
Others
HDFS
Directory
… rwx
… rwx
… rwx
Group D … rwx
Group F … rwx
User Y … rwx
© Hortonworks Inc. 2011
HDFS ACLs
New Tools for ACL Management (setfacl, getfacl)
– hdfs dfs -setfacl -m group:execs:r-- /sales-data
– hdfs dfs -getfacl /sales-data # file: /sales-data # owner: maya # group:
sales user::rw- group::r-- group:execs:r-- mask::r-- other::--
– How do you know if a directory has ACLs set?
– hdfs dfs -ls /sales-data Found 1 items -rw-r-----+ 3 maya sales 0
2014-03-04 16:31 /sales-data
© Hortonworks Inc. 2011
HDFS ACLs
Default ACLs
–hdfs dfs -setfacl -m default:group:execs:r-x /monthly-sales-data
–hdfs dfs -mkdir /monthly-sales-data/JAN
–hdfs dfs –getfacl /monthly-sales-data/JAN
– # file: /monthly-sales-data/JAN # owner: maya # group: sales user::rwx group::r-
x group:execs:r-x mask::r-x other::--- default:user::rwx default:group::r-
x default:group:execs:r-x default:mask::r-x default:other::---
© Hortonworks Inc. 2011
HDFS ACLs Best Practices
• Start with traditional HDFS permissions to implement most permission requirements.
• Define a smaller number of ACLs to handle exceptional cases.
• A file with an ACL incurs an additional cost in memory in the NameNode compared to a file that
has only traditional permissions.
Page 20
Architecting the Future of Big Data
© Hortonworks Inc. 2011
Tables and Hive
Architecting the Future of Big Data
Page 21
© Hortonworks Inc. 2011
Table ACLs – The Challenge and Solution
• Hive and Pig have traditionally offer full table access control via HDFS access control
• The challenge in column-level access control
– Hive and Pig queries are executed as Tez-based tasks that access the HDFS files directly
– HDFS does not have knowledge of columns (there are several file/table formats)
• Solution for Column level ACLs
– Let Hive server check and submit the query execution
– Let the table be accessible only by special user (“HiveServer”)
– But one has to restrict the UDFs and file formats
– Good news: Hive provides an authorization plugin to do this cleanly
• Use standard SQL permission constructs
– GRANT/REVOKE
• Store the ACLs in Hive Metastore instead of some external DB
• But what about Pig, there is no Pig server …
Page 22
Architecting the Future of Big Data
© Hortonworks Inc. 2011
Hive ATZ-NG – Architecture
HDFS
Metastore
HiveServer2
O/JDBC Beeline CLI
• ATZ-NG is called for O/JDBC & Beeline CLI
• Standard SQL GRANT / REVOKE for management
• Privilege to register UDF restricted to Admin user
• Policy integrated with Table/View life cycle
Storage Based Authorization Provider
Hive
CLI
OozieHue
PIG HCat
Ambari
0. Enable HiveATZ-NG
1. Authentication
UDFs
Protected – column level
Protected – table level
Restrict direct access to Metastore
Protect HDFS with Kerberos & HDFS ACL
ATZ-NG
2. Authorization
© Hortonworks Inc. 2011
What about MR/Pig
• Note there is no Pig/MR server to submit and check column ACLs
• Hence in the same cluster running Hive
–You can cannot give Pig similar access control
–If Pig/MR is important,
–Use coarse grained table level
–Or run Pig/MR as privileged uses with full table level access
Page 26
© Hortonworks Inc. 2011
Hive ATZ-NG Example
Page 28
© Hortonworks Inc. 2011
Scenario
• Objective: Share Product Management Roadmap securely
• Actors:
–Admin Role – Specified in hive-site
– Admin role controls role memberships
–Product Management Role
– Should be able to create, read all road map details.
– Members: Vinay Shukla, Tim Hall
–Engineering Role
– Should be able to read (see) all roadmap details
– Members: Kevin Minder, Larry McCay
Page 29
© Hortonworks Inc. 2011
Step 1: Admin role Creates Roles, Adds Users
1. CREATE ROLE PM;
1. CREATE ROLE ENG;
1. GRANT ROLE PM to user timhall with admin option;
1. GRANT ROLE PM to user vinayshukla;
1. GRANT ROLE ENG to user kevinminder with admin option;
1. GRANT ROLE ENG to user larrymccay;
© Hortonworks Inc. 2011
Step 2: Super-user Creates Tables/Views
create table hdp_hadoop_plans (
id int,
hadoop_roadmap string,
hdp_roadmap string
);
© Hortonworks Inc. 2011
Step 3: Users or Roles Assigned To Tables
1. GRANT ALL ON hdp_hadoop_plans TO ROLE PM;
1. GRANT SELECT ON hdp_hadoop_plans TO ROLE ENG;
© Hortonworks Inc. 2011
HBase Cell Level Authorization
• The HBase permissions model already supports ACLs defined at the namespace, table, column
family and column level.
– This is sufficient to meet many requirements
– This can be insufficient if a data model requires protection on individual rows/cells.
– Example: Medical data, each row representing a patient, may require customizing who can see an individual patient’s data,
and the social security number of each row may need further restriction.
Page 33
© Hortonworks Inc. 2011
HBase Cell Level Authorization
• Cell level authorization augments the permissions model by allowing ACLs specified on
individual cells.
– ACLs are now supported at the individual cell level.
– Individual operations may choose order of evaluation. Cell level ACLs may be evaluated last or first.
– Evaluating last is useful if the common case is access granted through table or column family ACLs, and cell level
ACLs define exceptions for denial.
– Evaluating first is useful if many users are granted access through cell level ACLs.
Page 34
Architecting the Future of Big Data
© Hortonworks Inc. 2011
HBase Cell Level Authorization
• Visibility labels
– Visibility expressions can be stored as metadata in a cell’s tag.
– A visibility expression consists of labels combined with boolean operators.
– E.g. (financial | strategy | research) & !newhire
– This means that a user must be labeled financial or strategy or research and not be a newhire in order to see the column.
– The mapping of users to their labels is pluggable. By default, a user’s labels are specified as authorizations in the
individual operation.
– HBase visibility labels were inspired by similar features in Apache Accumulo, and the model will look very familiar
to Accumulo users.
Page 35
Architecting the Future of Big Data
© Hortonworks Inc. 2011
What can be done today?
Authorization
Restrict access to
explicit data
Audit
Understand who did
what
Data Protection
Encrypt data at rest &
motion
Wire encryption
• In native Hadoop
• With Knox
• SSL for Rest (2.x)
File encryption
• Via MR file format
• 3rd Party encryption
tools for col level
encryption
• Native HDFS support
coming
Authentication
Who am I/prove it?
Control access to cluster.
© Hortonworks Inc. 2011
Wire Encryption – for data in motion
Page 37
• Hadoop client to DataNode is via Data Transfer Protocol
– HDFS client reads/writes to HDFS service over encrypted channel
– Configurable encryption strength
• ODBC/JDBC Client to HiveServer 2
– Encryption is via SASL Quality Of Protection
• Map to Reduce via shuffle
– Shuffle is over HTTP(S)
– Supports mutual authentication via SSL
– Host name verification enabled
• Rest Protocols
– SSL support
© Hortonworks Inc. 2011
Data at Rest
• Coming: HDFS encrypted file system currently under development in Apache
– https://issues.apache.org/jira/browse/HADOOP-10150
– https://issues.apache.org/jira/browse/HDFS-6134
Page 38
Architecting the Future of Big Data
© Hortonworks Inc. 2011
XA Secure
A Major step forward in Hadoop security
See Shaun Connolly’s Key note on Wednesday June 4
Architecting the Future of Big Data
Page 39
© Hortonworks Inc. 2011
Security in Hadoop with HDP + XA Secure
Authorization
Restrict access to
explicit data
Audit
Understand who
did what
Data Protection
Encrypt data at
rest & in motion
• Kerberos in native
Apache Hadoop
• HTTP/REST API
Secured with
Apache Knox
Gateway
• MapReduce Access Control Lists
• HDFS Permissions, HDFS ACL,
• Audit logs in with HDFS & MR
• Hive ATZ-NG
• Cell level access control in
Apache Accumulo
Authentication
Who am I/prove it?
• Wire encryption
in Hadoop
• Orchestrated
encryption with
3rd party tools
• HDFS, Hive and
Hbase
• Fine grain
access control
• RBAC
• Centralized
audit reporting
• Policy and
access history
• Future roadmap
• Strategy to be
finalized
HDP2.1XASecure
Centralized Security Administration
• As-Is, works with
current
authentication
methods
© Hortonworks Inc. 2011
Open Source?
•Yes XASecure technology will be open sourced
–Not just a Apache license where you are forced to get
the latest from Hortonworks
–But a full-fledged Apache Project that is truly open to the
community of developers and users
•See Shaun Connolly’s Keynote on Wednesday for
details
Page 41
Architecting the Future of Big Data
© Hortonworks Inc. 2011
Summary
• Very strong Authentication via Kerberos and Active directory
– Uses your organization's user DB and integrates to its group and role membership
– Supplemented by Hadoop tokens
– Note these are necessary due to delayed job execution after user logs-off
• Strong fine grained authentication with some recent improvements
– HDFS ACLs
– Hive – integrated via SQL model and Hive Metastore
– Note a hacked side addon
– HBase Cell Level Authorization
• Strong encryption support
– Wire
– Data
– Some improvements coming soon
• Every product has audit logs
• XASecure adds a major step forward
– Yes it will be open sourced as a Apache Project
Page 42
Architecting the Future of Big Data
© Hortonworks Inc. 2011
Thank you, Q&A
Page 43
Resource Location
Hortonworks Security Labs http://hortonworks.com/labs/security/
Apache Knox Project Page http://knox.incubator.apache.org/
HDFS ACLs Blog Post http://hortonworks.com/blog/hdfs-acls-fine-grained-permissions-hdfs-files-hadoop/
Encrypted File System
Development
https://issues.apache.org/jira/browse/HADOOP-10150
https://issues.apache.org/jira/browse/HDFS-6134
HBase Cell Level
Authorization
https://blogs.apache.org/hbase/entry/hbase_cell_security
Learn more

Mais conteúdo relacionado

Mais procurados

Data protection for hadoop environments
Data protection for hadoop environmentsData protection for hadoop environments
Data protection for hadoop environmentsDataWorks Summit
 
Best Practices for Virtualizing Hadoop
Best Practices for Virtualizing HadoopBest Practices for Virtualizing Hadoop
Best Practices for Virtualizing HadoopDataWorks Summit
 
Aman sharma hyd_12crac High Availability Day 2015
Aman sharma hyd_12crac High Availability Day 2015Aman sharma hyd_12crac High Availability Day 2015
Aman sharma hyd_12crac High Availability Day 2015aioughydchapter
 
Hadoop World 2011: Hadoop Gateway - Konstantin Schvako, eBay
Hadoop World 2011: Hadoop Gateway - Konstantin Schvako, eBayHadoop World 2011: Hadoop Gateway - Konstantin Schvako, eBay
Hadoop World 2011: Hadoop Gateway - Konstantin Schvako, eBayCloudera, Inc.
 
SUSE: Infraestructura definida por software para BigData
SUSE: Infraestructura definida por software para BigDataSUSE: Infraestructura definida por software para BigData
SUSE: Infraestructura definida por software para BigDataJuan Herrera Utande
 
Database-as-a-Service with Oracle Enterprise Manager Cloud Control 12c and Or...
Database-as-a-Service with Oracle Enterprise Manager Cloud Control 12c and Or...Database-as-a-Service with Oracle Enterprise Manager Cloud Control 12c and Or...
Database-as-a-Service with Oracle Enterprise Manager Cloud Control 12c and Or...Leighton Nelson
 
Upgrade Without the Headache: Best Practices for Upgrading Hadoop in Production
Upgrade Without the Headache: Best Practices for Upgrading Hadoop in ProductionUpgrade Without the Headache: Best Practices for Upgrading Hadoop in Production
Upgrade Without the Headache: Best Practices for Upgrading Hadoop in ProductionCloudera, Inc.
 
Running Enterprise Workloads in the Cloud
Running Enterprise Workloads in the CloudRunning Enterprise Workloads in the Cloud
Running Enterprise Workloads in the CloudDataWorks Summit
 
Webinar: Productionizing Hadoop: Lessons Learned - 20101208
Webinar: Productionizing Hadoop: Lessons Learned - 20101208Webinar: Productionizing Hadoop: Lessons Learned - 20101208
Webinar: Productionizing Hadoop: Lessons Learned - 20101208Cloudera, Inc.
 
Apache Falcon - Simplifying Managing Data Jobs on Hadoop
Apache Falcon - Simplifying Managing Data Jobs on HadoopApache Falcon - Simplifying Managing Data Jobs on Hadoop
Apache Falcon - Simplifying Managing Data Jobs on HadoopDataWorks Summit
 
MySQL & Oracle Linux Keynote at Open Source India 2014
MySQL & Oracle Linux Keynote at Open Source India 2014MySQL & Oracle Linux Keynote at Open Source India 2014
MySQL & Oracle Linux Keynote at Open Source India 2014Sanjay Manwani
 
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBegan
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBeganKoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBegan
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBeganTobias Koprowski
 
Security implementation on hadoop
Security implementation on hadoopSecurity implementation on hadoop
Security implementation on hadoopWei-Chiu Chuang
 
Melbourne Chef Meetup: Automating Azure Compliance with InSpec
Melbourne Chef Meetup: Automating Azure Compliance with InSpecMelbourne Chef Meetup: Automating Azure Compliance with InSpec
Melbourne Chef Meetup: Automating Azure Compliance with InSpecMatt Ray
 
eProseed Oracle Open World 2016 debrief - Oracle 12.2.0.1 Database
eProseed Oracle Open World 2016 debrief - Oracle 12.2.0.1 DatabaseeProseed Oracle Open World 2016 debrief - Oracle 12.2.0.1 Database
eProseed Oracle Open World 2016 debrief - Oracle 12.2.0.1 DatabaseMarco Gralike
 
Gunther hagleitner:apache hive & stinger
Gunther hagleitner:apache hive & stingerGunther hagleitner:apache hive & stinger
Gunther hagleitner:apache hive & stingerhdhappy001
 

Mais procurados (20)

Data protection for hadoop environments
Data protection for hadoop environmentsData protection for hadoop environments
Data protection for hadoop environments
 
Best Practices for Virtualizing Hadoop
Best Practices for Virtualizing HadoopBest Practices for Virtualizing Hadoop
Best Practices for Virtualizing Hadoop
 
Aman sharma hyd_12crac High Availability Day 2015
Aman sharma hyd_12crac High Availability Day 2015Aman sharma hyd_12crac High Availability Day 2015
Aman sharma hyd_12crac High Availability Day 2015
 
Hadoop World 2011: Hadoop Gateway - Konstantin Schvako, eBay
Hadoop World 2011: Hadoop Gateway - Konstantin Schvako, eBayHadoop World 2011: Hadoop Gateway - Konstantin Schvako, eBay
Hadoop World 2011: Hadoop Gateway - Konstantin Schvako, eBay
 
SUSE: Infraestructura definida por software para BigData
SUSE: Infraestructura definida por software para BigDataSUSE: Infraestructura definida por software para BigData
SUSE: Infraestructura definida por software para BigData
 
Database-as-a-Service with Oracle Enterprise Manager Cloud Control 12c and Or...
Database-as-a-Service with Oracle Enterprise Manager Cloud Control 12c and Or...Database-as-a-Service with Oracle Enterprise Manager Cloud Control 12c and Or...
Database-as-a-Service with Oracle Enterprise Manager Cloud Control 12c and Or...
 
Upgrade Without the Headache: Best Practices for Upgrading Hadoop in Production
Upgrade Without the Headache: Best Practices for Upgrading Hadoop in ProductionUpgrade Without the Headache: Best Practices for Upgrading Hadoop in Production
Upgrade Without the Headache: Best Practices for Upgrading Hadoop in Production
 
Running Enterprise Workloads in the Cloud
Running Enterprise Workloads in the CloudRunning Enterprise Workloads in the Cloud
Running Enterprise Workloads in the Cloud
 
Webinar: Productionizing Hadoop: Lessons Learned - 20101208
Webinar: Productionizing Hadoop: Lessons Learned - 20101208Webinar: Productionizing Hadoop: Lessons Learned - 20101208
Webinar: Productionizing Hadoop: Lessons Learned - 20101208
 
Apache Falcon - Simplifying Managing Data Jobs on Hadoop
Apache Falcon - Simplifying Managing Data Jobs on HadoopApache Falcon - Simplifying Managing Data Jobs on Hadoop
Apache Falcon - Simplifying Managing Data Jobs on Hadoop
 
Toward Better Multi-Tenancy Support from HDFS
Toward Better Multi-Tenancy Support from HDFSToward Better Multi-Tenancy Support from HDFS
Toward Better Multi-Tenancy Support from HDFS
 
MySQL & Oracle Linux Keynote at Open Source India 2014
MySQL & Oracle Linux Keynote at Open Source India 2014MySQL & Oracle Linux Keynote at Open Source India 2014
MySQL & Oracle Linux Keynote at Open Source India 2014
 
Apache Slider
Apache SliderApache Slider
Apache Slider
 
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBegan
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBeganKoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBegan
KoprowskiT_PASSEastMidsFEB16_2AMaDisasterJustBegan
 
Security implementation on hadoop
Security implementation on hadoopSecurity implementation on hadoop
Security implementation on hadoop
 
Hadoop Operations
Hadoop OperationsHadoop Operations
Hadoop Operations
 
SQL On Hadoop
SQL On HadoopSQL On Hadoop
SQL On Hadoop
 
Melbourne Chef Meetup: Automating Azure Compliance with InSpec
Melbourne Chef Meetup: Automating Azure Compliance with InSpecMelbourne Chef Meetup: Automating Azure Compliance with InSpec
Melbourne Chef Meetup: Automating Azure Compliance with InSpec
 
eProseed Oracle Open World 2016 debrief - Oracle 12.2.0.1 Database
eProseed Oracle Open World 2016 debrief - Oracle 12.2.0.1 DatabaseeProseed Oracle Open World 2016 debrief - Oracle 12.2.0.1 Database
eProseed Oracle Open World 2016 debrief - Oracle 12.2.0.1 Database
 
Gunther hagleitner:apache hive & stinger
Gunther hagleitner:apache hive & stingerGunther hagleitner:apache hive & stinger
Gunther hagleitner:apache hive & stinger
 

Destaque

Securing Hadoop's REST APIs with Apache Knox Gateway Hadoop Summit June 6th, ...
Securing Hadoop's REST APIs with Apache Knox Gateway Hadoop Summit June 6th, ...Securing Hadoop's REST APIs with Apache Knox Gateway Hadoop Summit June 6th, ...
Securing Hadoop's REST APIs with Apache Knox Gateway Hadoop Summit June 6th, ...Kevin Minder
 
Apache Hadoop Security - Ranger
Apache Hadoop Security - RangerApache Hadoop Security - Ranger
Apache Hadoop Security - RangerIsheeta Sanghi
 
Hdp security overview
Hdp security overview Hdp security overview
Hdp security overview Hortonworks
 
Discover Enterprise Security Features in Hortonworks Data Platform 2.1: Apach...
Discover Enterprise Security Features in Hortonworks Data Platform 2.1: Apach...Discover Enterprise Security Features in Hortonworks Data Platform 2.1: Apach...
Discover Enterprise Security Features in Hortonworks Data Platform 2.1: Apach...Hortonworks
 
Securing Your Apache Spark Applications
Securing Your Apache Spark ApplicationsSecuring Your Apache Spark Applications
Securing Your Apache Spark ApplicationsCloudera, Inc.
 
Spark Summit Keynote with Ken Tsai
Spark Summit Keynote with Ken TsaiSpark Summit Keynote with Ken Tsai
Spark Summit Keynote with Ken TsaiSpark Summit
 
Real-Time Anomoly Detection with Spark MLib, Akka and Cassandra by Natalino Busa
Real-Time Anomoly Detection with Spark MLib, Akka and Cassandra by Natalino BusaReal-Time Anomoly Detection with Spark MLib, Akka and Cassandra by Natalino Busa
Real-Time Anomoly Detection with Spark MLib, Akka and Cassandra by Natalino BusaSpark Summit
 
Petabyte Scale Anomaly Detection Using R & Spark by Sridhar Alla and Kiran Mu...
Petabyte Scale Anomaly Detection Using R & Spark by Sridhar Alla and Kiran Mu...Petabyte Scale Anomaly Detection Using R & Spark by Sridhar Alla and Kiran Mu...
Petabyte Scale Anomaly Detection Using R & Spark by Sridhar Alla and Kiran Mu...Spark Summit
 
The Future of Hadoop Security - Hadoop Summit 2014
The Future of Hadoop Security - Hadoop Summit 2014The Future of Hadoop Security - Hadoop Summit 2014
The Future of Hadoop Security - Hadoop Summit 2014Cloudera, Inc.
 
Case study of online machine learning for display advertising in Yahoo! JAPAN
Case study of online machine learning for display advertising in Yahoo! JAPANCase study of online machine learning for display advertising in Yahoo! JAPAN
Case study of online machine learning for display advertising in Yahoo! JAPANDataWorks Summit/Hadoop Summit
 
Hadoop Security Today & Tomorrow with Apache Knox
Hadoop Security Today & Tomorrow with Apache KnoxHadoop Security Today & Tomorrow with Apache Knox
Hadoop Security Today & Tomorrow with Apache KnoxVinay Shukla
 
Using Hadoop to build a Data Quality Service for both real-time and batch data
Using Hadoop to build a Data Quality Service for both real-time and batch dataUsing Hadoop to build a Data Quality Service for both real-time and batch data
Using Hadoop to build a Data Quality Service for both real-time and batch dataDataWorks Summit/Hadoop Summit
 
Anomaly Detection with Apache Spark
Anomaly Detection with Apache SparkAnomaly Detection with Apache Spark
Anomaly Detection with Apache SparkCloudera, Inc.
 
Enabling Apache Zeppelin and Spark for Data Science in the Enterprise
Enabling Apache Zeppelin and Spark for Data Science in the EnterpriseEnabling Apache Zeppelin and Spark for Data Science in the Enterprise
Enabling Apache Zeppelin and Spark for Data Science in the EnterpriseDataWorks Summit/Hadoop Summit
 
Spark and the Future of Advanced Analytics by Thomas Dinsmore
Spark and the Future of Advanced Analytics by Thomas DinsmoreSpark and the Future of Advanced Analytics by Thomas Dinsmore
Spark and the Future of Advanced Analytics by Thomas DinsmoreSpark Summit
 
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...DataWorks Summit/Hadoop Summit
 
Moving towards enterprise ready Hadoop clusters on the cloud
Moving towards enterprise ready Hadoop clusters on the cloudMoving towards enterprise ready Hadoop clusters on the cloud
Moving towards enterprise ready Hadoop clusters on the cloudDataWorks Summit/Hadoop Summit
 
Securing Hadoop with Apache Ranger
Securing Hadoop with Apache RangerSecuring Hadoop with Apache Ranger
Securing Hadoop with Apache RangerDataWorks Summit
 

Destaque (20)

Securing Hadoop's REST APIs with Apache Knox Gateway Hadoop Summit June 6th, ...
Securing Hadoop's REST APIs with Apache Knox Gateway Hadoop Summit June 6th, ...Securing Hadoop's REST APIs with Apache Knox Gateway Hadoop Summit June 6th, ...
Securing Hadoop's REST APIs with Apache Knox Gateway Hadoop Summit June 6th, ...
 
Apache Hadoop Security - Ranger
Apache Hadoop Security - RangerApache Hadoop Security - Ranger
Apache Hadoop Security - Ranger
 
Hdp security overview
Hdp security overview Hdp security overview
Hdp security overview
 
Securing Spark Applications
Securing Spark ApplicationsSecuring Spark Applications
Securing Spark Applications
 
Discover Enterprise Security Features in Hortonworks Data Platform 2.1: Apach...
Discover Enterprise Security Features in Hortonworks Data Platform 2.1: Apach...Discover Enterprise Security Features in Hortonworks Data Platform 2.1: Apach...
Discover Enterprise Security Features in Hortonworks Data Platform 2.1: Apach...
 
Securing Your Apache Spark Applications
Securing Your Apache Spark ApplicationsSecuring Your Apache Spark Applications
Securing Your Apache Spark Applications
 
Spark Summit Keynote with Ken Tsai
Spark Summit Keynote with Ken TsaiSpark Summit Keynote with Ken Tsai
Spark Summit Keynote with Ken Tsai
 
Real-Time Anomoly Detection with Spark MLib, Akka and Cassandra by Natalino Busa
Real-Time Anomoly Detection with Spark MLib, Akka and Cassandra by Natalino BusaReal-Time Anomoly Detection with Spark MLib, Akka and Cassandra by Natalino Busa
Real-Time Anomoly Detection with Spark MLib, Akka and Cassandra by Natalino Busa
 
Apache Ranger
Apache RangerApache Ranger
Apache Ranger
 
Petabyte Scale Anomaly Detection Using R & Spark by Sridhar Alla and Kiran Mu...
Petabyte Scale Anomaly Detection Using R & Spark by Sridhar Alla and Kiran Mu...Petabyte Scale Anomaly Detection Using R & Spark by Sridhar Alla and Kiran Mu...
Petabyte Scale Anomaly Detection Using R & Spark by Sridhar Alla and Kiran Mu...
 
The Future of Hadoop Security - Hadoop Summit 2014
The Future of Hadoop Security - Hadoop Summit 2014The Future of Hadoop Security - Hadoop Summit 2014
The Future of Hadoop Security - Hadoop Summit 2014
 
Case study of online machine learning for display advertising in Yahoo! JAPAN
Case study of online machine learning for display advertising in Yahoo! JAPANCase study of online machine learning for display advertising in Yahoo! JAPAN
Case study of online machine learning for display advertising in Yahoo! JAPAN
 
Hadoop Security Today & Tomorrow with Apache Knox
Hadoop Security Today & Tomorrow with Apache KnoxHadoop Security Today & Tomorrow with Apache Knox
Hadoop Security Today & Tomorrow with Apache Knox
 
Using Hadoop to build a Data Quality Service for both real-time and batch data
Using Hadoop to build a Data Quality Service for both real-time and batch dataUsing Hadoop to build a Data Quality Service for both real-time and batch data
Using Hadoop to build a Data Quality Service for both real-time and batch data
 
Anomaly Detection with Apache Spark
Anomaly Detection with Apache SparkAnomaly Detection with Apache Spark
Anomaly Detection with Apache Spark
 
Enabling Apache Zeppelin and Spark for Data Science in the Enterprise
Enabling Apache Zeppelin and Spark for Data Science in the EnterpriseEnabling Apache Zeppelin and Spark for Data Science in the Enterprise
Enabling Apache Zeppelin and Spark for Data Science in the Enterprise
 
Spark and the Future of Advanced Analytics by Thomas Dinsmore
Spark and the Future of Advanced Analytics by Thomas DinsmoreSpark and the Future of Advanced Analytics by Thomas Dinsmore
Spark and the Future of Advanced Analytics by Thomas Dinsmore
 
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...
 
Moving towards enterprise ready Hadoop clusters on the cloud
Moving towards enterprise ready Hadoop clusters on the cloudMoving towards enterprise ready Hadoop clusters on the cloud
Moving towards enterprise ready Hadoop clusters on the cloud
 
Securing Hadoop with Apache Ranger
Securing Hadoop with Apache RangerSecuring Hadoop with Apache Ranger
Securing Hadoop with Apache Ranger
 

Semelhante a Improvements in Hadoop Security

Hadoop Security Today and Tomorrow
Hadoop Security Today and TomorrowHadoop Security Today and Tomorrow
Hadoop Security Today and TomorrowDataWorks Summit
 
Storage and-compute-hdfs-map reduce
Storage and-compute-hdfs-map reduceStorage and-compute-hdfs-map reduce
Storage and-compute-hdfs-map reduceChris Nauroth
 
An Apache Hive Based Data Warehouse
An Apache Hive Based Data WarehouseAn Apache Hive Based Data Warehouse
An Apache Hive Based Data WarehouseDataWorks Summit
 
2014 sept 4_hadoop_security
2014 sept 4_hadoop_security2014 sept 4_hadoop_security
2014 sept 4_hadoop_securityAdam Muise
 
Securing Hadoop in an Enterprise Context
Securing Hadoop in an Enterprise ContextSecuring Hadoop in an Enterprise Context
Securing Hadoop in an Enterprise ContextHellmar Becker
 
Hw09 Security And Api Compatibility
Hw09   Security And Api CompatibilityHw09   Security And Api Compatibility
Hw09 Security And Api CompatibilityCloudera, Inc.
 
Plugging the Holes: Security and Compatability in Hadoop
Plugging the Holes: Security and Compatability in HadoopPlugging the Holes: Security and Compatability in Hadoop
Plugging the Holes: Security and Compatability in HadoopOwen O'Malley
 
Treat your enterprise data lake indigestion: Enterprise ready security and go...
Treat your enterprise data lake indigestion: Enterprise ready security and go...Treat your enterprise data lake indigestion: Enterprise ready security and go...
Treat your enterprise data lake indigestion: Enterprise ready security and go...DataWorks Summit
 
CIS13: Big Data Platform Vendor’s Perspective: Insights from the Bleeding Edge
CIS13: Big Data Platform Vendor’s Perspective: Insights from the Bleeding EdgeCIS13: Big Data Platform Vendor’s Perspective: Insights from the Bleeding Edge
CIS13: Big Data Platform Vendor’s Perspective: Insights from the Bleeding EdgeCloudIDSummit
 
Hadoop and Data Access Security
Hadoop and Data Access SecurityHadoop and Data Access Security
Hadoop and Data Access SecurityCloudera, Inc.
 
TriHUG October: Apache Ranger
TriHUG October: Apache RangerTriHUG October: Apache Ranger
TriHUG October: Apache Rangertrihug
 
Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho...
Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho...Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho...
Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho...huguk
 
Securing the Hadoop Ecosystem
Securing the Hadoop EcosystemSecuring the Hadoop Ecosystem
Securing the Hadoop EcosystemDataWorks Summit
 
Hadoop & Security - Past, Present, Future
Hadoop & Security - Past, Present, FutureHadoop & Security - Past, Present, Future
Hadoop & Security - Past, Present, FutureUwe Printz
 
Troubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the BeastTroubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the BeastDataWorks Summit
 
How to Upgrade Your Hadoop Stack in 1 Step -- with Zero Downtime
How to Upgrade Your Hadoop Stack in 1 Step -- with Zero DowntimeHow to Upgrade Your Hadoop Stack in 1 Step -- with Zero Downtime
How to Upgrade Your Hadoop Stack in 1 Step -- with Zero DowntimeIan Lumb
 
Introduction to the Hadoop EcoSystem
Introduction to the Hadoop EcoSystemIntroduction to the Hadoop EcoSystem
Introduction to the Hadoop EcoSystemShivaji Dutta
 
Bridle your Flying Islands and Castles in the Sky: Built-in Governance and Se...
Bridle your Flying Islands and Castles in the Sky: Built-in Governance and Se...Bridle your Flying Islands and Castles in the Sky: Built-in Governance and Se...
Bridle your Flying Islands and Castles in the Sky: Built-in Governance and Se...DataWorks Summit
 
A glimpse into the Future of Hadoop & Big Data
A glimpse into the Future of Hadoop & Big DataA glimpse into the Future of Hadoop & Big Data
A glimpse into the Future of Hadoop & Big DataSaurav Kumar Sinha
 

Semelhante a Improvements in Hadoop Security (20)

Hadoop Security Today and Tomorrow
Hadoop Security Today and TomorrowHadoop Security Today and Tomorrow
Hadoop Security Today and Tomorrow
 
Storage and-compute-hdfs-map reduce
Storage and-compute-hdfs-map reduceStorage and-compute-hdfs-map reduce
Storage and-compute-hdfs-map reduce
 
An Apache Hive Based Data Warehouse
An Apache Hive Based Data WarehouseAn Apache Hive Based Data Warehouse
An Apache Hive Based Data Warehouse
 
2014 sept 4_hadoop_security
2014 sept 4_hadoop_security2014 sept 4_hadoop_security
2014 sept 4_hadoop_security
 
Securing Hadoop in an Enterprise Context
Securing Hadoop in an Enterprise ContextSecuring Hadoop in an Enterprise Context
Securing Hadoop in an Enterprise Context
 
Hw09 Security And Api Compatibility
Hw09   Security And Api CompatibilityHw09   Security And Api Compatibility
Hw09 Security And Api Compatibility
 
Plugging the Holes: Security and Compatability in Hadoop
Plugging the Holes: Security and Compatability in HadoopPlugging the Holes: Security and Compatability in Hadoop
Plugging the Holes: Security and Compatability in Hadoop
 
Treat your enterprise data lake indigestion: Enterprise ready security and go...
Treat your enterprise data lake indigestion: Enterprise ready security and go...Treat your enterprise data lake indigestion: Enterprise ready security and go...
Treat your enterprise data lake indigestion: Enterprise ready security and go...
 
CIS13: Big Data Platform Vendor’s Perspective: Insights from the Bleeding Edge
CIS13: Big Data Platform Vendor’s Perspective: Insights from the Bleeding EdgeCIS13: Big Data Platform Vendor’s Perspective: Insights from the Bleeding Edge
CIS13: Big Data Platform Vendor’s Perspective: Insights from the Bleeding Edge
 
Hadoop and Data Access Security
Hadoop and Data Access SecurityHadoop and Data Access Security
Hadoop and Data Access Security
 
TriHUG October: Apache Ranger
TriHUG October: Apache RangerTriHUG October: Apache Ranger
TriHUG October: Apache Ranger
 
Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho...
Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho...Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho...
Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho...
 
Hadoop security
Hadoop securityHadoop security
Hadoop security
 
Securing the Hadoop Ecosystem
Securing the Hadoop EcosystemSecuring the Hadoop Ecosystem
Securing the Hadoop Ecosystem
 
Hadoop & Security - Past, Present, Future
Hadoop & Security - Past, Present, FutureHadoop & Security - Past, Present, Future
Hadoop & Security - Past, Present, Future
 
Troubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the BeastTroubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the Beast
 
How to Upgrade Your Hadoop Stack in 1 Step -- with Zero Downtime
How to Upgrade Your Hadoop Stack in 1 Step -- with Zero DowntimeHow to Upgrade Your Hadoop Stack in 1 Step -- with Zero Downtime
How to Upgrade Your Hadoop Stack in 1 Step -- with Zero Downtime
 
Introduction to the Hadoop EcoSystem
Introduction to the Hadoop EcoSystemIntroduction to the Hadoop EcoSystem
Introduction to the Hadoop EcoSystem
 
Bridle your Flying Islands and Castles in the Sky: Built-in Governance and Se...
Bridle your Flying Islands and Castles in the Sky: Built-in Governance and Se...Bridle your Flying Islands and Castles in the Sky: Built-in Governance and Se...
Bridle your Flying Islands and Castles in the Sky: Built-in Governance and Se...
 
A glimpse into the Future of Hadoop & Big Data
A glimpse into the Future of Hadoop & Big DataA glimpse into the Future of Hadoop & Big Data
A glimpse into the Future of Hadoop & Big Data
 

Mais de Chris Nauroth

Hadoop & cloud storage object store integration in production (final)
Hadoop & cloud storage  object store integration in production (final)Hadoop & cloud storage  object store integration in production (final)
Hadoop & cloud storage object store integration in production (final)Chris Nauroth
 
Keep your hadoop cluster at its best! v4
Keep your hadoop cluster at its best! v4Keep your hadoop cluster at its best! v4
Keep your hadoop cluster at its best! v4Chris Nauroth
 
Hdfs 2016-hadoop-summit-san-jose-v4
Hdfs 2016-hadoop-summit-san-jose-v4Hdfs 2016-hadoop-summit-san-jose-v4
Hdfs 2016-hadoop-summit-san-jose-v4Chris Nauroth
 
Hdfs 2016-hadoop-summit-dublin-v1
Hdfs 2016-hadoop-summit-dublin-v1Hdfs 2016-hadoop-summit-dublin-v1
Hdfs 2016-hadoop-summit-dublin-v1Chris Nauroth
 
Hadoop operations-2015-hadoop-summit-san-jose-v5
Hadoop operations-2015-hadoop-summit-san-jose-v5Hadoop operations-2015-hadoop-summit-san-jose-v5
Hadoop operations-2015-hadoop-summit-san-jose-v5Chris Nauroth
 
Hadoop operations-2014-strata-new-york-v5
Hadoop operations-2014-strata-new-york-v5Hadoop operations-2014-strata-new-york-v5
Hadoop operations-2014-strata-new-york-v5Chris Nauroth
 
Interactive Hadoop via Flash and Memory
Interactive Hadoop via Flash and MemoryInteractive Hadoop via Flash and Memory
Interactive Hadoop via Flash and MemoryChris Nauroth
 

Mais de Chris Nauroth (7)

Hadoop & cloud storage object store integration in production (final)
Hadoop & cloud storage  object store integration in production (final)Hadoop & cloud storage  object store integration in production (final)
Hadoop & cloud storage object store integration in production (final)
 
Keep your hadoop cluster at its best! v4
Keep your hadoop cluster at its best! v4Keep your hadoop cluster at its best! v4
Keep your hadoop cluster at its best! v4
 
Hdfs 2016-hadoop-summit-san-jose-v4
Hdfs 2016-hadoop-summit-san-jose-v4Hdfs 2016-hadoop-summit-san-jose-v4
Hdfs 2016-hadoop-summit-san-jose-v4
 
Hdfs 2016-hadoop-summit-dublin-v1
Hdfs 2016-hadoop-summit-dublin-v1Hdfs 2016-hadoop-summit-dublin-v1
Hdfs 2016-hadoop-summit-dublin-v1
 
Hadoop operations-2015-hadoop-summit-san-jose-v5
Hadoop operations-2015-hadoop-summit-san-jose-v5Hadoop operations-2015-hadoop-summit-san-jose-v5
Hadoop operations-2015-hadoop-summit-san-jose-v5
 
Hadoop operations-2014-strata-new-york-v5
Hadoop operations-2014-strata-new-york-v5Hadoop operations-2014-strata-new-york-v5
Hadoop operations-2014-strata-new-york-v5
 
Interactive Hadoop via Flash and Memory
Interactive Hadoop via Flash and MemoryInteractive Hadoop via Flash and Memory
Interactive Hadoop via Flash and Memory
 

Último

Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxJoão Esperancinha
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native BuildpacksVish Abrams
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyRaymond Okyere-Forson
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampVICTOR MAESTRE RAMIREZ
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?AmeliaSmith90
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptkinjal48
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfBrain Inventory
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 

Último (20)

Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptx
 
Salesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptxSalesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptx
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native Buildpacks
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human Beauty
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - Datacamp
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.ppt
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdf
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 

Improvements in Hadoop Security

  • 1. © Hortonworks Inc. 2011 Improvements in Hadoop Security Sanjay Radia sanjay@hortonworks.com @srr Chris Nauroth cnauroth@hortonworks.com @cnauroth Page 1
  • 2. © Hortonworks Inc. 2011 Hello Sanjay Radia • Founder, Hortonworks • Part of the Hadoop team at Yahoo! since 2007 – Chief Architect of Hadoop Core at Yahoo! – Long time Apache Hadoop PMC and Committer – Designed and developed several key Hadoop features • Prior – Data center automation, virtualization, Java, HA, OSs, File Systems (Startup, Sun Microsystems, …) – Ph.D., University of Waterloo Chris Nauroth • Member of Technical Staff, Hortonworks – Apache Hadoop Committer – Major contributor to HDFS ACLs • Hadoop user since 2010 – Prior employment experience deploying, maintaining and using Hadoop clusters Page 2 Architecting the Future of Big Data
  • 3. © Hortonworks Inc. 2011 Overview • Models of Deployment – Secure and insecure • Hadoop Authentication – The how and why – Knox – perimeter security • Authorization – existing and what is new – HDFS – Tables and Hive – HBase and Accumulo • Data protection and encryption – Wire – Data at rest Page 3 Architecting the Future of Big Data
  • 4. © Hortonworks Inc. 2011 Two Reasons for Security in Hadoop Hadoop Contains Sensitive Data – As Hadoop adoption grows so too has the types of data organizations look to store. Often the data is proprietary or personal and it must be protected. – In this context, Hadoop is governed by the same security requirements as any data center platform. Hadoop is subject to Compliance adherence – Organizations are often subject to comply with regulations such as HIPPA, PCI DSS, FISAM that require protection of personal information. – Adherence to other Corporate security policies. 1 2
  • 5. © Hortonworks Inc. 2011 Three Models of Hadoop Deployment • Insecure cluster – You have protected via the perimeter – You trust the code that runs in the system – - note In Hadoop cluster, user submitted code runs inside the cluster – (Note true in typical client-server applications) – The client side libraries pass the client’s login credential – There is not end-end-authentication here – Authorization is done against this credentials • Secure cluster – Full authentication – Can run arbitrary code in jobs • Perimeter security using Knox – Internal cluster can be secure or insecure depending on your needs
  • 6. © Hortonworks Inc. 2011 Pillars of Hadoop Security Authorization Restrict access to explicit data Audit Understand who did what Data Protection Encrypt data at rest & motion AD/Kerberos in native Apache Hadoop Perimeter Security with Apache Knox Gateway Authentication Who am I/prove it? Control access to cluster.
  • 7. © Hortonworks Inc. 2011 Hadoop Authentication Overview • Kerberos/Active Directory based security – SSO – users do not have to re-login into Hadoop – Hadoop accounts do not have to be recreated – Caveat – MR Task isolation require Unix accounts for each user current but this is going away with Linux containers – Hadoop tokens – supplement the Kerberos authentication – Delegation tokens – deal with the delayed job execution – Block tokens – capabilities to deal with the distributed nature of HDFS – Trusted Proxies – support for third party services to act as proxy – Oozie – Gateways – HDFS proxy, Knox, etc • Base security infrastructure being extended to support other authentication services besides Kerberos • Knox – Perimeter Security and Rest Gateway Page 7
  • 8. © Hortonworks Inc. 2011 User DB/Account Single-signon using your organization’s existing user DB • No need to create Hadoop user accounts to be created – Note the OS account on compute nodes are for isolation – this will be removed when we use Linux containers • Currently supports Kerberos (and LDAP indirectly) • The RPC layer’s authentication is fairly general – Supports Kerberos and Hadoop’s tokens – Can be extended
  • 9. © Hortonworks Inc. 2011 Why the tokens • Why does Hadoop have its own tokens? – Standard client-server security model is not sufficient for Hadoop – Works when logged-in client is directly accessing a Hadoop service – But for a job, the execution happens much later – The job submitter has long logged off • Hence we needed to add delegation tokens • HDFS is a distributed service and needed to add capability-like tokens for datanode authentication The permissions/ ACLs are in the Namenode
  • 10. © Hortonworks Inc. 2011 Apache Knox Perimeter Security for Hadoop REST APIs Architecting the Future of Big Data Page 10
  • 11. © Hortonworks Inc. 2011 The Gateway or Edge Node • Hadoop APIs can be used from any desktop after SSO login – FileSystem and MapReduce Java APIs – Pig, Hive and Oozie clients (that wrap the Java APIs) • However it is typical to Use “Edge Node” or “Gateway Node” that is “inside” cluster – The libraries for the APIs are generally only installed on the gateway – Users SSH to Edge Node and execute API commands from shell Page 11 HadoopUser Edge Node SSH
  • 12. © Hortonworks Inc. 2011 • Single Hadoop access point • REST API hierarchy • Consolidated API calls • Multi-cluster support • Eliminates SSH “edge node” • Central API management • Central audit control • Simple Service level Authorization • SSO Integration – Siteminder, API Key*, OAuth* & SAML* • LDAP & AD integration Perimeter Security with Apache Knox Integrated with existing systems to simplify identity maintenance Incubated and led by Hortonworks, Apache Knox provides a simple and open framework for Hadoop perimeter security. Single, simple point of access for a cluster Central controls ensure consistency across one or more clusters
  • 13. © Hortonworks Inc. 2011 Hadoop REST APIs • Useful for connecting to Hadoop from the outside the cluster • When more client language flexibility is required – i.e. Java binding not an option • Challenges (Knox addresses these challenges) – Client must have knowledge of cluster topology – Required to open ports (and in some cases, on every host) outside the cluster Page 13 Service API WebHDFS Supports HDFS user operations including reading files, writing to files, making directories, changing permissions and renaming. Learn more about WebHDFS. WebHCat Job control for MapReduce, Pig and Hive jobs, and HCatalog DDL commands. Learn more about WebHCat. Hive Hive REST API operations HBase HBase REST API operations Oozie Job submission and management, and Oozie administration. Learn more about Oozie.
  • 14. © Hortonworks Inc. 2011 What can be done today? Authorization Restrict access to explicit data Audit Understand who did what Data Protection Encrypt data at rest & motion Previously • All Services: Service level ACLs • HDFS: Permissions • Yarn: Queue ACLs • Hive/Pig Tables: Table level via HDFS • Apache Accumulo: Cell level • HBase: Namespace, Table, Column Family and Column level ACLs Authentication Who am I/prove it? Control access to cluster. Hadoop 2.x • HDFS: ACLs • Hive: Column level ACLs • HBase: Cell level ACLs • Knox: • Rest Service level Authorization • Access Audit with Knox
  • 15. © Hortonworks Inc. 2011 HDFS ACLs • Existing HDFS POSIX permissions good, but not flexible enough – Permission requirements may differ from the natural organizational hierarchy of users and groups. • HDFS ACLs augment the existing HDFS POSIX permissions model by implementing the POSIX ACL model. – An ACL (Access Control List) provides a way to set different permissions for specific named users or named groups, not only the file’s owner and file’s group. Page 15 Architecting the Future of Big Data
  • 16. © Hortonworks Inc. 2011 HDFS File Permissions Example • Authorization requirements: –In a sales department, they would like a single user Maya (Department Manager) to control all modifications to sales data –Other members of sales department need to view the data, but can’t modify it. –Everyone else in the company must not be allowed to view the data. • Can be implemented via the following: Read/Write perm for user maya User Group Read perm for group sales File with sales data
  • 17. © Hortonworks Inc. 2011 HDFS ACLs • Problem –No longer feasible for Maya to control all modifications to the file – New Requirement: Maya, Diane and Clark are allowed to make modifications – New Requirement: New group called executives should be able to read the sales data –Current permissions model only allows permissions at 1 group and 1 user • Solution: HDFS ACLs –Now assign different permissions to different users and groups Owner Group Others HDFS Directory … rwx … rwx … rwx Group D … rwx Group F … rwx User Y … rwx
  • 18. © Hortonworks Inc. 2011 HDFS ACLs New Tools for ACL Management (setfacl, getfacl) – hdfs dfs -setfacl -m group:execs:r-- /sales-data – hdfs dfs -getfacl /sales-data # file: /sales-data # owner: maya # group: sales user::rw- group::r-- group:execs:r-- mask::r-- other::-- – How do you know if a directory has ACLs set? – hdfs dfs -ls /sales-data Found 1 items -rw-r-----+ 3 maya sales 0 2014-03-04 16:31 /sales-data
  • 19. © Hortonworks Inc. 2011 HDFS ACLs Default ACLs –hdfs dfs -setfacl -m default:group:execs:r-x /monthly-sales-data –hdfs dfs -mkdir /monthly-sales-data/JAN –hdfs dfs –getfacl /monthly-sales-data/JAN – # file: /monthly-sales-data/JAN # owner: maya # group: sales user::rwx group::r- x group:execs:r-x mask::r-x other::--- default:user::rwx default:group::r- x default:group:execs:r-x default:mask::r-x default:other::---
  • 20. © Hortonworks Inc. 2011 HDFS ACLs Best Practices • Start with traditional HDFS permissions to implement most permission requirements. • Define a smaller number of ACLs to handle exceptional cases. • A file with an ACL incurs an additional cost in memory in the NameNode compared to a file that has only traditional permissions. Page 20 Architecting the Future of Big Data
  • 21. © Hortonworks Inc. 2011 Tables and Hive Architecting the Future of Big Data Page 21
  • 22. © Hortonworks Inc. 2011 Table ACLs – The Challenge and Solution • Hive and Pig have traditionally offer full table access control via HDFS access control • The challenge in column-level access control – Hive and Pig queries are executed as Tez-based tasks that access the HDFS files directly – HDFS does not have knowledge of columns (there are several file/table formats) • Solution for Column level ACLs – Let Hive server check and submit the query execution – Let the table be accessible only by special user (“HiveServer”) – But one has to restrict the UDFs and file formats – Good news: Hive provides an authorization plugin to do this cleanly • Use standard SQL permission constructs – GRANT/REVOKE • Store the ACLs in Hive Metastore instead of some external DB • But what about Pig, there is no Pig server … Page 22 Architecting the Future of Big Data
  • 23. © Hortonworks Inc. 2011 Hive ATZ-NG – Architecture HDFS Metastore HiveServer2 O/JDBC Beeline CLI • ATZ-NG is called for O/JDBC & Beeline CLI • Standard SQL GRANT / REVOKE for management • Privilege to register UDF restricted to Admin user • Policy integrated with Table/View life cycle Storage Based Authorization Provider Hive CLI OozieHue PIG HCat Ambari 0. Enable HiveATZ-NG 1. Authentication UDFs Protected – column level Protected – table level Restrict direct access to Metastore Protect HDFS with Kerberos & HDFS ACL ATZ-NG 2. Authorization
  • 24. © Hortonworks Inc. 2011 What about MR/Pig • Note there is no Pig/MR server to submit and check column ACLs • Hence in the same cluster running Hive –You can cannot give Pig similar access control –If Pig/MR is important, –Use coarse grained table level –Or run Pig/MR as privileged uses with full table level access Page 26
  • 25. © Hortonworks Inc. 2011 Hive ATZ-NG Example Page 28
  • 26. © Hortonworks Inc. 2011 Scenario • Objective: Share Product Management Roadmap securely • Actors: –Admin Role – Specified in hive-site – Admin role controls role memberships –Product Management Role – Should be able to create, read all road map details. – Members: Vinay Shukla, Tim Hall –Engineering Role – Should be able to read (see) all roadmap details – Members: Kevin Minder, Larry McCay Page 29
  • 27. © Hortonworks Inc. 2011 Step 1: Admin role Creates Roles, Adds Users 1. CREATE ROLE PM; 1. CREATE ROLE ENG; 1. GRANT ROLE PM to user timhall with admin option; 1. GRANT ROLE PM to user vinayshukla; 1. GRANT ROLE ENG to user kevinminder with admin option; 1. GRANT ROLE ENG to user larrymccay;
  • 28. © Hortonworks Inc. 2011 Step 2: Super-user Creates Tables/Views create table hdp_hadoop_plans ( id int, hadoop_roadmap string, hdp_roadmap string );
  • 29. © Hortonworks Inc. 2011 Step 3: Users or Roles Assigned To Tables 1. GRANT ALL ON hdp_hadoop_plans TO ROLE PM; 1. GRANT SELECT ON hdp_hadoop_plans TO ROLE ENG;
  • 30. © Hortonworks Inc. 2011 HBase Cell Level Authorization • The HBase permissions model already supports ACLs defined at the namespace, table, column family and column level. – This is sufficient to meet many requirements – This can be insufficient if a data model requires protection on individual rows/cells. – Example: Medical data, each row representing a patient, may require customizing who can see an individual patient’s data, and the social security number of each row may need further restriction. Page 33
  • 31. © Hortonworks Inc. 2011 HBase Cell Level Authorization • Cell level authorization augments the permissions model by allowing ACLs specified on individual cells. – ACLs are now supported at the individual cell level. – Individual operations may choose order of evaluation. Cell level ACLs may be evaluated last or first. – Evaluating last is useful if the common case is access granted through table or column family ACLs, and cell level ACLs define exceptions for denial. – Evaluating first is useful if many users are granted access through cell level ACLs. Page 34 Architecting the Future of Big Data
  • 32. © Hortonworks Inc. 2011 HBase Cell Level Authorization • Visibility labels – Visibility expressions can be stored as metadata in a cell’s tag. – A visibility expression consists of labels combined with boolean operators. – E.g. (financial | strategy | research) & !newhire – This means that a user must be labeled financial or strategy or research and not be a newhire in order to see the column. – The mapping of users to their labels is pluggable. By default, a user’s labels are specified as authorizations in the individual operation. – HBase visibility labels were inspired by similar features in Apache Accumulo, and the model will look very familiar to Accumulo users. Page 35 Architecting the Future of Big Data
  • 33. © Hortonworks Inc. 2011 What can be done today? Authorization Restrict access to explicit data Audit Understand who did what Data Protection Encrypt data at rest & motion Wire encryption • In native Hadoop • With Knox • SSL for Rest (2.x) File encryption • Via MR file format • 3rd Party encryption tools for col level encryption • Native HDFS support coming Authentication Who am I/prove it? Control access to cluster.
  • 34. © Hortonworks Inc. 2011 Wire Encryption – for data in motion Page 37 • Hadoop client to DataNode is via Data Transfer Protocol – HDFS client reads/writes to HDFS service over encrypted channel – Configurable encryption strength • ODBC/JDBC Client to HiveServer 2 – Encryption is via SASL Quality Of Protection • Map to Reduce via shuffle – Shuffle is over HTTP(S) – Supports mutual authentication via SSL – Host name verification enabled • Rest Protocols – SSL support
  • 35. © Hortonworks Inc. 2011 Data at Rest • Coming: HDFS encrypted file system currently under development in Apache – https://issues.apache.org/jira/browse/HADOOP-10150 – https://issues.apache.org/jira/browse/HDFS-6134 Page 38 Architecting the Future of Big Data
  • 36. © Hortonworks Inc. 2011 XA Secure A Major step forward in Hadoop security See Shaun Connolly’s Key note on Wednesday June 4 Architecting the Future of Big Data Page 39
  • 37. © Hortonworks Inc. 2011 Security in Hadoop with HDP + XA Secure Authorization Restrict access to explicit data Audit Understand who did what Data Protection Encrypt data at rest & in motion • Kerberos in native Apache Hadoop • HTTP/REST API Secured with Apache Knox Gateway • MapReduce Access Control Lists • HDFS Permissions, HDFS ACL, • Audit logs in with HDFS & MR • Hive ATZ-NG • Cell level access control in Apache Accumulo Authentication Who am I/prove it? • Wire encryption in Hadoop • Orchestrated encryption with 3rd party tools • HDFS, Hive and Hbase • Fine grain access control • RBAC • Centralized audit reporting • Policy and access history • Future roadmap • Strategy to be finalized HDP2.1XASecure Centralized Security Administration • As-Is, works with current authentication methods
  • 38. © Hortonworks Inc. 2011 Open Source? •Yes XASecure technology will be open sourced –Not just a Apache license where you are forced to get the latest from Hortonworks –But a full-fledged Apache Project that is truly open to the community of developers and users •See Shaun Connolly’s Keynote on Wednesday for details Page 41 Architecting the Future of Big Data
  • 39. © Hortonworks Inc. 2011 Summary • Very strong Authentication via Kerberos and Active directory – Uses your organization's user DB and integrates to its group and role membership – Supplemented by Hadoop tokens – Note these are necessary due to delayed job execution after user logs-off • Strong fine grained authentication with some recent improvements – HDFS ACLs – Hive – integrated via SQL model and Hive Metastore – Note a hacked side addon – HBase Cell Level Authorization • Strong encryption support – Wire – Data – Some improvements coming soon • Every product has audit logs • XASecure adds a major step forward – Yes it will be open sourced as a Apache Project Page 42 Architecting the Future of Big Data
  • 40. © Hortonworks Inc. 2011 Thank you, Q&A Page 43 Resource Location Hortonworks Security Labs http://hortonworks.com/labs/security/ Apache Knox Project Page http://knox.incubator.apache.org/ HDFS ACLs Blog Post http://hortonworks.com/blog/hdfs-acls-fine-grained-permissions-hdfs-files-hadoop/ Encrypted File System Development https://issues.apache.org/jira/browse/HADOOP-10150 https://issues.apache.org/jira/browse/HDFS-6134 HBase Cell Level Authorization https://blogs.apache.org/hbase/entry/hbase_cell_security Learn more

Notas do Editor

  1. Note: I would have used Tom Reilly instead of Mike Olson but nobody knows who Tom is.
  2. WebHDFS and WebHcat surely support SSL – others check?