Graph of Enterprise Metadata Topic | Knowledge Graphs - Graph of Enterprise Metadata When | 2020-10-20 1:35 PM PST Video | https://youtu.be/8flUr0J9rkU?t=137 Live Stream | https://www.bigmarker.com/neo4j-event/Mastering-Enterprise-Metadata-with-Neo4j?bmid=116f67e48bad Conference | https://neo4j.com/nodes-2020/agenda/

1. Graph of Enterprise Metadata–
Powered by Neo4J & Spark
Vladimir Bacvanski
Deepak Chandramouli

2. https://neo4j.com/nodes-2020/

3. AGENDA
 Prelude
 Introduction
 Architecture & Flow
 Scaling for enterprise
 Future
 Questions

4. About us
Vladimir Bacvanski
vbacvanski@paypal.com
Twitter: @OnSoftware
• Principal Architect, Strategic Architecture at PayPal
• In previous life: CTO of a development and
consulting firm
• PhD in Computer Science from RWTH Aachen,
Germany
• O’Reilly author: Courses on Big Data, Kafka
Deepak Chandramouli
dmohanakumarchan@paypal.com
LinkedIn: @deepakmc
• MT2 Software Engineer, Data Platform Services at
PayPal
• Data Enthusiast
• Tech lead
• Gimel (Big Data Framework for Apache Spark)
• Unified Data Catalog – PayPal’s Enterprise Data
Catalog

5. Prelude

6. 2019 | Recommendation System – Unified Data Catalog
Building Recommendations for a Data Catalog
A
B
S
T
R
A
C
T
I
O
N
L
A
Y
E
R
Yeah, let’s
expand on
this !
Graph of
metadata
has much
more to
Offer !!

7. Graph of Enterprise Metadata
Introduction

8. The Enterprise
Landscape
8
On-Premise
Adjacencies
Data Technologies
Cloud Services
Multi-DC / Hybrid Cloud
Org Structures CollaborationPeople
Identity & Access Security Zones Data Catalog
Security
Organization
Artifacts Apps & Processes
Logs
Streams
ETL Apps

9. Connecting the Dots…
Effective
Data
Governance
Privacy
Regulation
Complianc
e
……
The whole is
greater ! Efficiency.
Connected
Components
Security
Organizatio
n
Data
Business
MetadataTechnical
Metadata
Artifacts
Code
Operational
Metadata
Internal
Social
Metadata
Data
Classification
(PII/PCI/PHI)

10. Connected components = Graph of Enterprise Metadata

11. So What  Top Down View !
©2016 PayPal Inc. Confidential and proprietary.

12. Demo & Sandbox Code

13. Code Base | Playground
Github
https://github.com/Dee-Pac/GEM
Gitter
https://gitter.im/graph_of_enterprise_metadata/GEM-Ask-Us-
Anything

14. Graph Build
Architecture & Data Flow

15. Metadata Data Sources...
• Data Classification
• PII / PCI / PHI
• Operational Metadata
• Jobs & Apps
• Databases & App – Query Logs
• Org Structure
• People, Geo , Hierarchies
• Technical Data Catalog
• Data Stores
• Databases / Containers
• Tables / Objects
• Columns / Fields
• Business Metadata / Glossary / Annotation
• IAM
• Access controls
• Owners, Users
• Social Metadata (Internal)
• Slack Chats, Threads, Annotations
• Artifacts
• Code base – GitHub/SQL
• Wiki – Documents, Mentions
• JIRA – Issues, Mentions

16. High Level – Data Flow
Merge Graph
CRUD &
Insights
App Data Sources
Data Sourcing
Transform
Application
s
Access Simplification
Distributed Processing for scale
(Future) Near Real Time - CDC
Graph Access
API Push

17. neo4j + Apache Spark + Gimel + GraphQL
Multiple Sources Systems
• Abstracts Graph Complexity
• Supports - Multiple Front-end
clients
• Accelerates UI Integration
• Reference | neo4j & GraphQL
• Spring-Data-neo4j [Scale]
Unified Access PatternData Processing
• Distributed Processing -
SCALE
• Spark-ML + Graphx. -
COMPLEXITY
• GIMEL + Spark - Data
access simplified
• Kafka Streams – Near Real
Time CDC in the future
Connected Components
• Easy Bootstrap
• Cypher & APOCs
• Robust libraries - Spark,
GraphQL
• Rich Graph Algorithms

18. <<<<<< Data Processing @ Scale
>>>>>

19. © 2020 PayPal Inc. Confidential and proprietary.
Technical Metadata
>1000s of Data Stores
>100K Containers
>7 Million Datasets
>50 Million Fields
>10K PII/PCI Class elements
Operational Metadata
>100K Data Apps Per Day
>25 Data Tech Stacks
>API, Streams & Batch Apps
Artifacts & Social
Metadata,
Artifacts
>10K Scripts & App code
>1000s of Wiki, JIRA Annotations
Data Social
~ 12 Active Analytics Channel in Slack
~ Evolving Teams conversations for data
Security & Organization
>1000’s of IAM roles
>20K Internal Users & Role Owners
Metadata Growth

20. Gimel + spark + neo4j
©2016 PayPal Inc. Confidential and proprietary.
GIMEL Data API
NEO4J Connector

21. Gimel + spark + neo4j (Continued..)
Unified Data API
Unified Connector Config

22. Unified Data API
Gimel + spark + neo4j (Continued..)
Nodes &
Relationship

23. Exposing Graph to Clients

24. GraphQL Query
GraphQL Query - Response
type column {
column:String!
storage_system_name:String!
dataset:String!
}
type dataset {
dataset:String!
storage_system_name:String!
column: [column]
@relation(name:"DATASET_HAS_COLUMN",
direction:OUT)
}
type Enterprise {
name:String!
}
Define GraphQL Schema
Cypher -> GraphQL
Cypher

25. Conclusion & Next Steps

26. Key Take-aways
• The whole is greater than Sum !
• Data Awareness: Data can be anywhere across the ecosystem : but we should know its shape & form of existence.
• Graphs are everywhere: Full potential can be realized by connecting the dots - across board.
• Enriching metadata: It is a collective effort; silos rarely achieve effectiveness.
• Graph’s Effectiveness: expose the data in a consumable way for – all tech stacks alike !

27. What’s next?
• Data Classification
• PII / PCI / PHI
• Org Structure
• People, Geo , Hierarchies
• Technical Data Catalog
• Data Stores
• Databases / Containers
• Tables / Objects
• Columns / Fields
• Business Metadata / Glossary / Annotation
• IAM
• Access controls
• Owners, Users
• Artifacts
• Code base – GitHub/SQL
• Wiki – Documents, Mentions
• JIRA – Issues, Mentions
• Operational Metadata
• Jobs & Apps
• Databases & App – Query Logs
• Lineage
• Social Metadata (Internal)
• Slack Chats, Threads,
Annotations
Connecting Further…
Scale – Millions of Nodes X Edges
Access - Simplification
Graph – Data Security &
Multitenancy
Insights driven by –
Graph / ML Algorithms
Foundational Capabilities…

28. Questions?
Code base – GITHUB_FOR_GEM
Let’s collaborate - GITTER_FOR_GEM

29. Thank You!