Notes on Data Governance in Hadoop. This is my self learning by reading hortonworks and cloudera manual, then come up with these slides. Please refer in detail at cloudera.com

1. Data Governance In Hadoop
1
Dr. Komes Chandavimol
Hortonwork & Cloudera

2. Traditional Database Architecture
2

3. Modern Database Architecture
3

4. Modern Database Architecture
4

5. Modern Database Architecture
5

6. 6

7. Modern Database Architecture
7

8. Data Governance
8
https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.3.6
Data governance in HDP is managed by the following components:
Apache Falcon: solves enterprise challenges related to Hadoop data replication, business
continuity, and lineage tracing by deploying a framework for data management and processing.
The Falcon framework can also leverage other HDP components, such as Pig, HDFS, and Oozie.
Falcon enables this simplified management by providing a framework to define, deploy, and
manage data pipelines. Data pipelines contain:
• A definition of the dataset to be processed.
• Interfaces to the Hadoop cluster where the data resides.
• A process definition that defines how the data is consumed and invokes processing
logic.
Apache Atlas: extends Falcon's governance capabilities by adding business taxonomical and
operational metadata. Atlas is a scalable and extensible set of core governance services that
enable enterprises to meet their compliance requirements within the Hadoop stack and to
integrate with their data ecosystem outside HDP. Atlas provides:
• Data classification.
• Centralized auditing.
• Search and lineage history.
• Security and policy engines.

9. Apache Falcon
Apache Falcon addresses the following data governance requirements and provides a
wizard-like GUI that eliminates hand coding of complex data sets and offers:
Centrally manage the data lifecycle: Falcon enables you to manage the data lifecyle in one
common place where you can define and manage policies and pipelines for data ingest,
processing, and export.
Business continuity and disaster recovery: Falcon can replicate HDFS and Hive datasets,
trigger processes for retry, and handle late data arrival logic. In addition, Falcon can mirror
file systems or Hive HCatalog on clusters using recipes that enable to you re-use complex
workflows.
Address audit and compliance requirements: Falcon provides audit and compliance
features that enable you to visualize data pipeline lineage, track data pipeline audit logs,
and tag data with business metadata.
9
https://hadoopsters.com/2016/02/08/guide-to-apache-falcon-6-monitoring-jobs/

10. Apache Falcon
10
Falcon can be installed and managed by Apache Ambari, and jobs can be traced through the
native Falcon UI.
Falcon can process data from:
• Oozie jobs
• Pig scripts
• Hive scripts
https://hadoopsters.com/2016/02/08/guide-to-apache-falcon-6-monitoring-jobs/

11. Apache Falcon
11
https://hadoopsters.com/2016/02/08/guide-to-apache-falcon-6-monitoring-jobs/
Falcon Architecture

12. Apache Falcon
12
https://docs.cloudera.com/HDPDocuments/HDP2/HDP-
2.3.6/bk_data_governance/content/ch_falcon_basic_concepts.html
Data Pipeline

13. Apache Falcon
13
https://hadoopsters.com/2016/02/08/guide-to-apache-falcon-6-monitoring-jobs/
Falcon Architecture

14. Apache Ambari
14

15. Apache Atlas
15
https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.3.6/bk_data_governance/content/ch_hdp_data_governance_overview.html
Apache Atlas is a low-level service in the Hadoop stack that provides core metadata services.
Initially, Atlas provides metadata services for Hive, but in subsequent releases all components of
HDP will be brought under Atlas metadata management.
Knowledge store that leverages existing Hadoop Metastores: Categorized into a business-oriented
taxonomy of data sets, objects, tables, and columns. Supports the exchange of metadata between HDP
foundation components and third-party applications or governance tools.
Data lifecycle management: Leverages existing investment in Apache Falcon with a focus on
provenance, multi-cluster replication, data set retention and eviction, late data handling, and
automation.
Audit store: Historical repository for all governance events, including security events (access, grant,
deny), operational events related to data provenance and metrics. The Atlas audit store is indexed and
searchable for access to governance events.
Security: Integration with HDP security that enables you to establish global security policies based on
data classifications and that leverages Apache Ranger plug-in architecture for security policy
enforcement.
Policy engine: Fully extensible policy engine that supports metadata-based, geo-based, and time-based
rules that rationalize at runtime.
RESTful interface: Supports extensibility by way of REST APIs to third-party applications so you can use
your existing tools to view and manipulate metadata in the HDP foundation components.

16. Apache Atlas
16
https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.3.6/bk_data_governance/content/ch_hdp_data_governance_overview.html
Apache Atlas is a low-level service in the Hadoop stack that provides core metadata services.
Initially, Atlas provides metadata services for Hive, but in subsequent releases all components of
HDP will be brought under Atlas metadata management.
Atlas provides:
• Metadata types & instances
• Classification
• Lineage
• Search/Discovery
• Security & Data Masking

17. Apache Atlas
17
https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.3.6/bk_data_governance/content/section_metadata_store_mgmt_with_atlas_ui.html

18. Apache Atlas – Metadata + Lineage
18
https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.3.6/bk_data_governance/content/section_metadata_store_mgmt_with_atlas_ui.html

19. Apache Atlas – Metadata + Lineage
19
https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.3.6/bk_data_governance/content/section_metadata_store_mgmt_with_atlas_ui.html

20. Apache Atlas – Dashboard
20
https://www2.slideshare.net/HadoopSummit/apache-atlas-governance-for-your-data

21. Apache Atlas – Classification
21
https://www2.slideshare.net/HadoopSummit/apache-atlas-governance-for-your-data

22. Apache Atlas – Classification
22
https://www2.slideshare.net/HadoopSummit/apache-atlas-governance-for-your-data

23. Apache Atlas – Classification
23
https://www2.slideshare.net/HadoopSummit/apache-atlas-governance-for-your-data

24. Apache Atlas – Classification
24
https://www2.slideshare.net/HadoopSummit/apache-atlas-governance-for-your-data

25. Apache Atlas – Classification
25
https://www2.slideshare.net/HadoopSummit/apache-atlas-governance-for-your-data

26. Apache Atlas – Catalog Search
26
https://www2.slideshare.net/HadoopSummit/apache-atlas-governance-for-your-data

27. Apache Ranger
27

28. Apache Ranger
28
https://www2.slideshare.net/HadoopSummit/security-and-data-governance-using-apache-ranger-and-apache-atlas

29. Apache Ranger
29
https://www2.slideshare.net/HadoopSummit/security-and-data-governance-using-apache-ranger-and-apache-atlas

30. Apache Ranger
30
https://www2.slideshare.net/HadoopSummit/security-and-data-governance-using-apache-ranger-and-apache-atlas

31. Apache Ranger
31
https://www2.slideshare.net/HadoopSummit/security-and-data-governance-using-apache-ranger-and-apache-atlas

32. Apache Ranger
32
https://www2.slideshare.net/HadoopSummit/security-and-data-governance-using-apache-ranger-and-apache-atlas
Ranger Architecture

33. Apache Ranger
33
https://www2.slideshare.net/HadoopSummit/security-and-data-governance-using-apache-ranger-and-apache-atlas

34. Apache Ranger
34
https://www2.slideshare.net/HadoopSummit/security-and-data-governance-using-apache-ranger-and-apache-atlas

35. Apache Ranger
35
https://www2.slideshare.net/HadoopSummit/security-and-data-governance-using-apache-ranger-and-apache-atlas

36. Apache Ranger
36
https://www2.slideshare.net/HadoopSummit/security-and-data-governance-using-apache-ranger-and-apache-atlas

37. Data Science and Engineering Platform
37
https://blog.cloudera.com/data-science-engineering-platform-hdp-3-0-hybrid-secure-scalable/

38. Data Science and Engineering Platform
38
https://blog.cloudera.com/data-science-engineering-platform-hdp-3-0-hybrid-secure-scalable/
Lineage for Data and Applications

39. Data Science and Engineering Platform
39
Deep Learning
An Autonomous Car Following a Lane
see the video
https://blog.cloudera.com/data-science-engineering-platform-hdp-3-0-hybrid-secure-scalable/

40. Cloudera Navigator
Cloudera Navigator is the only complete data governance solution for Hadoop, offering critical capabilities
such as data discovery, continuous optimization, audit, lineage, metadata management, and policy
enforcement. As part of Cloudera Enterprise, Cloudera Navigator is critical to enabling high-performance
agile analytics, supporting continuous data architecture optimization, and meeting regulatory compliance
requirements.
40
https://www.cloudera.com/products/product-components/cloudera-navigator.html

41. Cloudera Navigator - Search
41
https://www.cloudera.com/products/product-components/cloudera-navigator.html

42. Cloudera Navigator - Auditing
42
https://www.cloudera.com/products/product-components/cloudera-navigator.html

43. Cloudera Navigator - Auditing
43
https://www.cloudera.com/products/product-components/cloudera-navigator.html

44. Cloudera Navigator - Auditing
44
https://www.cloudera.com/products/product-components/cloudera-navigator.html

45. Cloudera Navigator – Analytics
45
Data Stewardship Dashboard
https://www.cloudera.com/products/product-components/cloudera-navigator.html

46. Cloudera Navigator – Analytics
46
Data Stewardship Dashboard
https://www.cloudera.com/products/product-components/cloudera-navigator.html

47. Cloudera Navigator – Analytics
47
Data Stewardship Dashboard
https://www.cloudera.com/products/product-components/cloudera-navigator.html

48. Cloudera Navigator – Analytics
48
Data Stewardship Dashboard
https://www.cloudera.com/products/product-components/cloudera-navigator.html

49. Cloudera Navigator – Policy
49
https://www.cloudera.com/products/product-components/cloudera-navigator.html
Data Stewardship Dashboard

50. Cloudera Navigator - Lineage
50
https://www.cloudera.com/products/product-components/cloudera-navigator.html

51. Cloudera Navigator - Metadata
51
https://www.cloudera.com/products/product-components/cloudera-navigator.html

52. Cloudera Navigator - Metadata
52
https://www.cloudera.com/products/product-components/cloudera-navigator.html

53. Apache Zeppelin
53

54. A New Era - Hortonworks and Cloudera
54
https://www.youtube.com/watch?v=nfFrEKFwQxs

55. Cloudera and Hortonwork
55

56. Cloudera Console
56

57. Cloudera Console
57

58. Cloudera Console
58

59. Cloudera and Hortonwork
59

60. Cloudera Data Platform
60

61. 61
https://www.youtube.com/watch?v=l-VU2UmN0_c

62. Cloudera Machine Learning
62
https://blog.cloudera.com/introducing-mlops-and-sdx-for-models-in-cloudera-machine-learning/

63. Cloudera Machine Learning
63
https://blog.cloudera.com/introducing-mlops-and-sdx-for-models-in-cloudera-machine-learning/

64. Cloudera Machine Learning
64
https://blog.cloudera.com/introducing-mlops-and-sdx-for-models-in-cloudera-machine-learning/

65. Cloudera Machine Learning
65
https://blog.cloudera.com/introducing-mlops-and-sdx-for-models-in-cloudera-machine-learning/

66. Reference
66