Security and Data Governance using Apache Ranger and Apache Atlas
•Transferir como PPTX, PDF•
3 gostaram•7,628 visualizações
This document discusses Apache Ranger and Apache Atlas for security and governance in Hadoop. It provides an overview of Ranger's centralized authorization and auditing capabilities for Hadoop components using policies. It also describes Atlas' capabilities for metadata management, data lineage, classification using tags, and integrations with Ranger for classification-based security. The document concludes with a demo and Q&A section.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Security and Data Governance using Apache Ranger and Apache Atlas
Semelhante a Security and Data Governance using Apache Ranger and Apache Atlas (20)
Mais de DataWorks Summit/Hadoop Summit
Mais de DataWorks Summit/Hadoop Summit (20)
Último
Último (20)
Security and Data Governance using Apache Ranger and Apache Atlas
- 1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Security & Governance using Apache Ranger & Apache Atlas October 2016 Madhan Neethiraj Director - Engineering, Security & Governance
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Disclaimer This document may contain product features and technology directions that are under development, may be under development in the future or may ultimately not be developed. Project capabilities are based on information that is publicly available within the Apache Software Foundation project websites ("Apache"). Progress of the project capabilities can be tracked from inception to release through Apache, however, technical feasibility, market demand, user feedback and the overarching Apache Software Foundation community development process can all effect timing and final delivery. This document’s description of these features and technology directions does not represent a contractual commitment, promise or obligation from Hortonworks to deliver these features in any generally available product. Product features and technology directions are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Since this document contains an outline of general product development plans, customers should not rely upon it when making purchasing decisions.
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • Introduction • Apache Ranger • Overview • Authorization policies • Row-filter, Column-masking policies • Audit logs • Apache Atlas • Overview • Lineage • Classification • Demo • Q & A
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Ranger: Overview Centralized authorization and auditing across Hadoop components • HDFS, Hive, HBase, Knox, Strom, YARN, Kafka, Solr, .. • Audit logs to: Solr, HDFS, Log4j, .. Access Authorization based on Resources, Resource Classification • Policies for specific set of resources – like a Hive database/table/column • Policies for resource classifications – like PII, PHI, PCI Row-filter, Column-masking based on policies • Restrict the rows accessible in a table based on users/groups/runtime-context • example: restrict users to access customer records for specific regions only • Mask or anonymize sensitive columns based on users/groups/runtime-context • example: only last 4 digits of account number should be available to few user-groups Extensible Architecture • Custom policy conditions, context enrichers • Easy to enable Ranger authorization and auditing for new components Encryption keys management to support Transparent Data Encryption
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Ranger: Centralized Administration Single pane of glass for security administration across multiple Hadoop components
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Ranger: Authorization Policies Consistent authorization policy structure across Hadoop components HDFS Resources Users/Groups/Permissions Hive Resources Users/Groups/Permissions
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Ranger: Row-filter, Column-masking Policies Row Filter to apply Mask to apply
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Ranger: Tag-based Policies Pick the tag Deny access to data after expiry date with the exception of ‘admin’ user
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Ranger: Access Audit Logs • Apache Ranger Plugins generate detailed audit logs of access to protected resources • Audit logs to multiple destinations: Solr, HDFS, Log4j appender • Interactive view of audit logs in Apache Ranger admin console
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Ranger: Architecture HBase Ranger Administration Portal HDFS Hive Server2 Ranger Audit Server Ranger Plugin HadoopComponentsEnterprise Users Legacy Tools and Data Governance Knox Ranger Policy Server Storm Solr HDFS Ranger Plugin Ranger Plugin Ranger Plugin Ranger Plugin Solr YARN Kafka Ranger Plugin NiFi Atlas Ranger Plugin Ranger Plugin Ranger Plugin Ranger Plugin Ranger UgSync Ranger TagSync LDAP/AD/OS Atlas
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Atlas: Introduction Metadata Repository • Flexible type system to capture schema/metadata of multiple components • Out-of-box models for Hive, HDFS, Storm, Falcon, Sqoop Data Lineage/Provenance • Captures data lineage across components Classification • Use tags to classify the data – like PII, PHI, PCI, EXPIRES_ON • Support for attributes in tags – like expiry_date Search • Search using classifications, attributes • Advanced search using DSL; convenient full-text search Integrations • With Apache Hive, Apache Storm, Apache Falcon, Apache Sqoop for metadata and lineage • With Apache Ranger for classification based security APIs to add support for more components
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Atlas: Lineage
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Atlas: Classification
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Atlas: Architecture
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Demo
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Questions
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved References
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved References • Apache Atlas • http://atlas.apache.org • http://hortonworks.com/apache/atlas • Apache Ranger • http://ranger.apache.org • http://hortonworks.com/apache/ranger • Apache Ranger wiki • https://cwiki.apache.org/confluence/display/RANGER • Tag based policies • https://cwiki.apache.org/confluence/display/RANGER/Tag+Based+Policies • Row-filtering and column-masking policies • https://cwiki.apache.org/confluence/display/RANGER/Row-level+filtering+and+column- masking+using+Apache+Ranger+policies+in+Apache+Hive
Notas do Editor
- We have a lot to cover, want to apologize in advance
- The Ranger Admin portal is the central interface for security administration. Users can create and update policies, which are then stored in a policy database. Plugins within each component poll these policies at regular intervals. The portal also consists of an audit server that sends audit data collected from the plugins for storage in HDFS or in a relational database. Ranger plugins: Plugins are lightweight Java programs which embed within processes of each cluster component. For example, the Apache Ranger plugin for Apache Hive is embedded within Hiveserver2.These plugins pull in policies from a central server and store them locally in a file. When a user request comes through the component, these plugins intercept the request and evaluate it against the security policy. Plugins also collect data from the user request and follow a separate thread to send this data back to the audit server. User group sync: Apache Ranger provides a user synchronization utility to pull users and groups from Unix or from LDAP or Active Directory. The user or group information is stored within Ranger portal and used for policy definition