SlideShare uma empresa Scribd logo
1 de 68
Baixar para ler offline
Mark Grover | Lyft | @mark_grover
Alyssa Ransbury | Square | @alyssaran
Slides: cutt.ly/a6n
Amundsen: From Discovering to Securing Data
Who we are
• Product Manager at Lyft
• Committer/PMC on various
Apache projects
• Previously developer on
Spark at Cloudera
2
• Security Engineer at Square
• Leads and supports on privacy
engineering efforts across
dozens of product teams
Problem
3
>35% consumer time spent on discovering & validating
trustworthy data
4
Analyst/DS workflow and time spent on each step
Other side effects
1. Interrupt heavy culture
● What data should I use for
X?
● Is that trustworthy?
2. Increased DB load
SELECT * … LIMIT 100
SELECT COUNT(*) over x days
Requirements &
solutions
5
1. What kind of information? (aka ABC of metadata)
6
Application Context
Metadata needed by humans or applications to operate
● Where is the data?
● What are the semantics of the data?
Behavior
How is data created and used over time?
● Who’s using the data?
● Who created the data?
Change
Change in data over time
● How is the data evolving over time?
● Evolution of code that generates the data
Terminology borrowed from Ground paper
Data Lineage TODAY
Short answer: Any data within your organization
Long answer:
2. About what data?
7
Data stores
Schema registry
Events /
Schemas
StreamsPeople
Employees
TODAY
NotebooksDashboard /
Reports
Processes
Develop breadth of applications
8
Metadata
Data
Discovery
& Trust
Compliance
(GDPR /
CCPA /
Financial)
Security /
ACL
Data
Monitoring
/ Ops
Data
Quality
Cost
Mgmt
Data
Mainten-
ance
Search based Lineage based Network based Programmatic
Where is the
table/dashboard for X?
What does it contain?
I am changing a data model,
who are the owner and most
common users?
I want to follow a
power user in my team.
Access metadata
programmatically
Does this analysis
already exist?
This table’s delivery was
delayed today, I want to
notify everyone downstream.
I want to bookmark
tables of interest and
get a feed of data
delay, schema change,
incidents.
Put (pull / push)
metadata
programmatically
Other requirements
● Leverage as much data automatically as possible
● Preferably, open source and healthy community
● Easy to set up
Goal: Reduce time to find trusted data w/ versatile graph
Meet Amundsen
10
First person to discover the South Pole -
Norwegian explorer, Roald Amundsen
Search for data, or browse
Search for datasets
See details of the data set
See detailed descriptions and profile of the column
Preview existing data, if perms allow
Search for existing dashboards (aka reports)
16
Dashboard detail page
17
Search for co-workers!
Search for data owned and used by your peers!
Impact
20
21
“This is God’s
work” - Head of
Analytics, Lyft
“I was on call and
I’m confident 50%
of the questions
could have been
answered by a
simple search in
Amundsen” - Data
Scientist,Lyft
A6n @ Lyft: 750+ WAUs, 150k+ tables, 4k+ employee pages
Roles of Amundsen users at Lyft
22
Penetration rate:
DS (aka analyst): 81%
RS (aka DS): 71%
PM: 22%
SWE: 17%
Cust Serv: 7% (12/390)
Sp. Ops: 67% (10/15)
Sp. Op Leads: 53% (9/17)
Economist: 100% (7/7)
Cust. Quality: 78% (7/9)
Growth Mktg: 25% (6/24)
Roadmap
23
Roadmap (subject to change, not ordered)
• Tighter Lineage integration / visualization
• Better view integration
• ACL integration, allow only specific roles to edit descriptions
• Show search context for what matched
• Index more resources (notebooks, Kafka topics, etc.)
24
Amundsen Open Source
25
700
Community
members
150+
Companies in
the community
20+
Companies using
in production
Amundsen Open Source Community
26
ProminentusersActivecommunity
Develop breadth of applications
27
Metadata
Data
Discovery
& Trust
Compliance
(GDPR /
CCPA /
Financial)
Security /
ACL
Data
Monitoring
/ Ops
Data
Quality
Cost
Mgmt
Data
Mainten-
ance
Square:
Scaling Data Security /
Privacy
28
29
Postgres Hive Redshift ... Presto
Github
Source
File
Databuilder Crawler
Neo4j
Elastic
Search
Metadata Service Search Service
Frontend ServiceML
Feature
Service
Security
Service
Other Microservices
Metadata Sources
30
Postgres Hive Redshift ... Presto
Github
Source
File
Databuilder Crawler
Neo4j
Elastic
Search
Metadata Service Search Service
Frontend ServiceML
Feature
Service
Security
Service
Other Microservices
Metadata Sources
31
Postgres Hive Redshift ... Presto
Github
Source
File
Databuilder Crawler
Neo4j
Elastic
Search
Metadata Service Search Service
Frontend ServiceML
Feature
Service
Security
Service
Other Microservices
Metadata Sources
32
Postgres Hive Redshift ... Presto
Github
Source
File
Databuilder Crawler
Neo4j
Elastic
Search
Metadata Service Search Service
Frontend ServiceML
Feature
Service
Security
Service
Other Microservices
Metadata Sources
33
Postgres Hive Redshift ... Presto
Github
Source
File
Databuilder Crawler
Neo4j
Elastic
Search
Metadata Service Search Service
Frontend ServiceML
Feature
Service
Security
Service
Other Microservices
Metadata Sources
34
The graph
Benefits of a graph
The Problem
36
● Relied mostly on manual work to understand our data
● We have A LOT of data, and growing
● NOT scalable
● Lots of goals related to data privacy on top of requirements from laws like
GDPR / CCPA
37
Postgres Hive Redshift ... Presto
Github
Source
File
Databuilder Crawler
Neo4j
Elastic
Search
Metadata Service Search Service
Frontend ServiceML
Feature
Service
Security
Service
Other Microservices
Metadata Sources
1. Store more
specific
metadata
2. Automate
data
inventorying
3. Support
more granular
access
controls
1. Store more specific
metadata
38
39
Postgres Hive Redshift ... Presto
Github
Source
File
Databuilder Crawler
Neo4j
Elastic
Search
Metadata Service Search Service
Frontend ServiceML
Feature
Service
Security
Service
Other Microservices
Metadata Sources
1. Store more
specific
metadata
New Metadata Types
40
PII
Semantic
Type
Data
Storage
Security
Data
Subject
Type
PII
Semantic
Type
for data in
Column
PII Semantic Type
41
A set value describing the general contents of data in a column, with a focus on
discovering data that fits into one of three buckets
● Sensitive by itself
● Could be sensitive when taken together with other data
● Links to sensitive data, like an internal identification token
Some
Column
Data
Data Storage Security
42
A set value describing the form of data in a column
PII
Semantic
Type
for data in
Column
Some
Column
Data
Data
Storage
Security
for data in
Column
Data Subject Type
43
A set value describing users whose information exists in a column
● Example: square_product_user
PII
Semantic
Type
for data in
Column
Some
Column
Data
Data
Storage
Security
for data in
Column
Data
Subject
Type
for data in
Column
Example
44
PII
Semantic
Type
Column 1
Data
Storage
Security
encrypted
Data
Subject
Type
app_customer
Data
Storage
Security
plaintext
Column 2
UI Changes
45
2. Automate data
inventorying*
46
* where possible
47
Postgres Hive Redshift ... Presto
Github
Source
File
Databuilder Crawler
Neo4j
Elastic
Search
Metadata Service Search Service
Frontend ServiceML
Feature
Service
Security
Service
Other Microservices
Metadata Sources
2. Automate
data
inventorying
Why?
● We have a lot of data!
● Metadata should be kept current at scale
● New data sources get created all the time
● Mistakes happen
48
Flagging Possibly Sensitive Data Locations
49
Some
Column
Data
Data
Storage
Security
encrypted
Flagging Possibly Sensitive Data Locations
50
Some
Column
Data
Data
Storage
Security
encrypted
Flag
Flags
Contain information about:
● The metadata type it relates to (i.e. PII Semantic Type, Data Storage Security)
● The possible value (i.e. encrypted for Data Storage Security)
● Human readable description
51
Some
Column
Data
Flag
Reason
Result
Reviewer
Checking for Data Sensitivity
● Column name
○ Keyword partial or full match
○ Literal PII match
● Column type
● Table name
● Sample values + Google DLP API
52
User Flow
53
Some
Column
Data
Flag
Data Storage
Security
plaintext
Flag
PII Semantic
Type
phone number
User Flow
54
Some
Column
Data
Flag
Data Storage
Security
plaintext
Result: True
Reviewer: person 1
Data
Storage
Security
plaintext
Flag
PII Semantic
Type
phone number
Result: False
Reviewer: person 2
UI Changes
55
UI Changes
56
3. Support more granular
access controls
57
58
Postgres Hive Redshift ... Presto
Github
Source
File
Databuilder Crawler
Neo4j
Elastic
Search
Metadata Service Search Service
Frontend ServiceML
Feature
Service
Security
Service
Other Microservices
Metadata Sources
3. Support
more granular
access
controls
Capability-governed access control
● Purpose - An enumerated, legal-approved reason to access specific PII
Semantic Types
● Capability - A combination of Purpose and Data Subject Type
59
Capability-governed access control
60
Capability
Purpose
Subject
Type
Semantic
Type
Semantic
Type
Capability-governed access control
61
Capability
001
USER SUPPORT
CASH
APP
CUSTOMER
PERSON
NAME
EMAIL
ADDRESS
Capability-governed access control
62
Amundsen
Gating
Service
4. Future Work
63
Future and Work in Progress
● Gremlin Proxy
● UI components
○ show users exactly the data they have access to based on their capabilities
○ show the data that is tied to a specific capability
○ empower data owners to revoke broad access to some data
● Supporting additional data types
● And beyond! (we are hiring…)
64
Summary
65
Develop breadth of applications
66
Metadata
Data
Discovery
& Trust
Compliance
(GDPR /
CCPA /
Financial)
Security /
ACL
Data
Monitoring
/ Ops
Data
Quality
Cost
Mgmt
Data
Mainten-
ance
Get started!
Check out Github:
github.com/lyft/amundsen
Join Slack (~700 users):
Link to join on github
67
Check out other blogs and videos:
Amundsen for Privacy @ Square
Introducing Amundsen
Amundsen's architecture
YouTube channel
Thanks!
68
Mark Grover | Lyft | @mark_grover
Alyssa Ransbury | Square | @alyssaran
Icons under Creative Commons License from https://thenounproject.com/

Mais conteúdo relacionado

Mais procurados

Sparkによる GISデータを題材とした時系列データ処理 (Hadoop / Spark Conference Japan 2016 講演資料)
Sparkによる GISデータを題材とした時系列データ処理 (Hadoop / Spark Conference Japan 2016 講演資料)Sparkによる GISデータを題材とした時系列データ処理 (Hadoop / Spark Conference Japan 2016 講演資料)
Sparkによる GISデータを題材とした時系列データ処理 (Hadoop / Spark Conference Japan 2016 講演資料)Hadoop / Spark Conference Japan
 
Apache Sparkの基本と最新バージョン3.2のアップデート(Open Source Conference 2021 Online/Fukuoka ...
Apache Sparkの基本と最新バージョン3.2のアップデート(Open Source Conference 2021 Online/Fukuoka ...Apache Sparkの基本と最新バージョン3.2のアップデート(Open Source Conference 2021 Online/Fukuoka ...
Apache Sparkの基本と最新バージョン3.2のアップデート(Open Source Conference 2021 Online/Fukuoka ...NTT DATA Technology & Innovation
 
Hadoopの概念と基本的知識
Hadoopの概念と基本的知識Hadoopの概念と基本的知識
Hadoopの概念と基本的知識Ken SASAKI
 
Maps and Meaning: Graph-based Entity Resolution in Apache Spark & GraphX
Maps and Meaning: Graph-based Entity Resolution in Apache Spark & GraphXMaps and Meaning: Graph-based Entity Resolution in Apache Spark & GraphX
Maps and Meaning: Graph-based Entity Resolution in Apache Spark & GraphXDatabricks
 
Introduction to Knowledge Graphs and Semantic AI
Introduction to Knowledge Graphs and Semantic AIIntroduction to Knowledge Graphs and Semantic AI
Introduction to Knowledge Graphs and Semantic AISemantic Web Company
 
PostgreSQLのリカバリ超入門(もしくはWAL、CHECKPOINT、オンラインバックアップの仕組み)
PostgreSQLのリカバリ超入門(もしくはWAL、CHECKPOINT、オンラインバックアップの仕組み)PostgreSQLのリカバリ超入門(もしくはWAL、CHECKPOINT、オンラインバックアップの仕組み)
PostgreSQLのリカバリ超入門(もしくはWAL、CHECKPOINT、オンラインバックアップの仕組み)Hironobu Suzuki
 
フックを使ったPostgreSQLの拡張機能を作ってみよう!(第33回PostgreSQLアンカンファレンス@オンライン 発表資料)
フックを使ったPostgreSQLの拡張機能を作ってみよう!(第33回PostgreSQLアンカンファレンス@オンライン 発表資料)フックを使ったPostgreSQLの拡張機能を作ってみよう!(第33回PostgreSQLアンカンファレンス@オンライン 発表資料)
フックを使ったPostgreSQLの拡張機能を作ってみよう!(第33回PostgreSQLアンカンファレンス@オンライン 発表資料)NTT DATA Technology & Innovation
 
Spark SQL Bucketing at Facebook
 Spark SQL Bucketing at Facebook Spark SQL Bucketing at Facebook
Spark SQL Bucketing at FacebookDatabricks
 
Engineering data quality
Engineering data qualityEngineering data quality
Engineering data qualityLars Albertsson
 
Hive Bucketing in Apache Spark with Tejas Patil
Hive Bucketing in Apache Spark with Tejas PatilHive Bucketing in Apache Spark with Tejas Patil
Hive Bucketing in Apache Spark with Tejas PatilDatabricks
 
スキーマレスカラムナフォーマット「Yosegi」で実現する スキーマの柔軟性と処理性能を両立したログ収集システム / Hadoop / Spark Con...
スキーマレスカラムナフォーマット「Yosegi」で実現する スキーマの柔軟性と処理性能を両立したログ収集システム / Hadoop / Spark Con...スキーマレスカラムナフォーマット「Yosegi」で実現する スキーマの柔軟性と処理性能を両立したログ収集システム / Hadoop / Spark Con...
スキーマレスカラムナフォーマット「Yosegi」で実現する スキーマの柔軟性と処理性能を両立したログ収集システム / Hadoop / Spark Con...Yahoo!デベロッパーネットワーク
 
VLDB2015 会議報告
VLDB2015 会議報告VLDB2015 会議報告
VLDB2015 会議報告Yuto Hayamizu
 
AWS Redshift Analyzeの必要性とvacuumの落とし穴
AWS Redshift Analyzeの必要性とvacuumの落とし穴AWS Redshift Analyzeの必要性とvacuumの落とし穴
AWS Redshift Analyzeの必要性とvacuumの落とし穴Moto Fukao
 
NAND Flash から InnoDB にかけての話(仮)
NAND Flash から InnoDB にかけての話(仮)NAND Flash から InnoDB にかけての話(仮)
NAND Flash から InnoDB にかけての話(仮)Takanori Sejima
 
40分でわかるHadoop徹底入門 (Cloudera World Tokyo 2014 講演資料)
40分でわかるHadoop徹底入門 (Cloudera World Tokyo 2014 講演資料) 40分でわかるHadoop徹底入門 (Cloudera World Tokyo 2014 講演資料)
40分でわかるHadoop徹底入門 (Cloudera World Tokyo 2014 講演資料) hamaken
 
Data Science Across Data Sources with Apache Arrow
Data Science Across Data Sources with Apache ArrowData Science Across Data Sources with Apache Arrow
Data Science Across Data Sources with Apache ArrowDatabricks
 
Speed up UDFs with GPUs using the RAPIDS Accelerator
Speed up UDFs with GPUs using the RAPIDS AcceleratorSpeed up UDFs with GPUs using the RAPIDS Accelerator
Speed up UDFs with GPUs using the RAPIDS AcceleratorDatabricks
 
問合せ最適化インサイド
問合せ最適化インサイド問合せ最適化インサイド
問合せ最適化インサイドTakahiro Itagaki
 
HIVE: Data Warehousing & Analytics on Hadoop
HIVE: Data Warehousing & Analytics on HadoopHIVE: Data Warehousing & Analytics on Hadoop
HIVE: Data Warehousing & Analytics on HadoopZheng Shao
 

Mais procurados (20)

Sparkによる GISデータを題材とした時系列データ処理 (Hadoop / Spark Conference Japan 2016 講演資料)
Sparkによる GISデータを題材とした時系列データ処理 (Hadoop / Spark Conference Japan 2016 講演資料)Sparkによる GISデータを題材とした時系列データ処理 (Hadoop / Spark Conference Japan 2016 講演資料)
Sparkによる GISデータを題材とした時系列データ処理 (Hadoop / Spark Conference Japan 2016 講演資料)
 
Apache Sparkの基本と最新バージョン3.2のアップデート(Open Source Conference 2021 Online/Fukuoka ...
Apache Sparkの基本と最新バージョン3.2のアップデート(Open Source Conference 2021 Online/Fukuoka ...Apache Sparkの基本と最新バージョン3.2のアップデート(Open Source Conference 2021 Online/Fukuoka ...
Apache Sparkの基本と最新バージョン3.2のアップデート(Open Source Conference 2021 Online/Fukuoka ...
 
Hadoopの概念と基本的知識
Hadoopの概念と基本的知識Hadoopの概念と基本的知識
Hadoopの概念と基本的知識
 
Maps and Meaning: Graph-based Entity Resolution in Apache Spark & GraphX
Maps and Meaning: Graph-based Entity Resolution in Apache Spark & GraphXMaps and Meaning: Graph-based Entity Resolution in Apache Spark & GraphX
Maps and Meaning: Graph-based Entity Resolution in Apache Spark & GraphX
 
Introduction to Knowledge Graphs and Semantic AI
Introduction to Knowledge Graphs and Semantic AIIntroduction to Knowledge Graphs and Semantic AI
Introduction to Knowledge Graphs and Semantic AI
 
PostgreSQLのリカバリ超入門(もしくはWAL、CHECKPOINT、オンラインバックアップの仕組み)
PostgreSQLのリカバリ超入門(もしくはWAL、CHECKPOINT、オンラインバックアップの仕組み)PostgreSQLのリカバリ超入門(もしくはWAL、CHECKPOINT、オンラインバックアップの仕組み)
PostgreSQLのリカバリ超入門(もしくはWAL、CHECKPOINT、オンラインバックアップの仕組み)
 
フックを使ったPostgreSQLの拡張機能を作ってみよう!(第33回PostgreSQLアンカンファレンス@オンライン 発表資料)
フックを使ったPostgreSQLの拡張機能を作ってみよう!(第33回PostgreSQLアンカンファレンス@オンライン 発表資料)フックを使ったPostgreSQLの拡張機能を作ってみよう!(第33回PostgreSQLアンカンファレンス@オンライン 発表資料)
フックを使ったPostgreSQLの拡張機能を作ってみよう!(第33回PostgreSQLアンカンファレンス@オンライン 発表資料)
 
Spark SQL Bucketing at Facebook
 Spark SQL Bucketing at Facebook Spark SQL Bucketing at Facebook
Spark SQL Bucketing at Facebook
 
Engineering data quality
Engineering data qualityEngineering data quality
Engineering data quality
 
Hive Bucketing in Apache Spark with Tejas Patil
Hive Bucketing in Apache Spark with Tejas PatilHive Bucketing in Apache Spark with Tejas Patil
Hive Bucketing in Apache Spark with Tejas Patil
 
スキーマレスカラムナフォーマット「Yosegi」で実現する スキーマの柔軟性と処理性能を両立したログ収集システム / Hadoop / Spark Con...
スキーマレスカラムナフォーマット「Yosegi」で実現する スキーマの柔軟性と処理性能を両立したログ収集システム / Hadoop / Spark Con...スキーマレスカラムナフォーマット「Yosegi」で実現する スキーマの柔軟性と処理性能を両立したログ収集システム / Hadoop / Spark Con...
スキーマレスカラムナフォーマット「Yosegi」で実現する スキーマの柔軟性と処理性能を両立したログ収集システム / Hadoop / Spark Con...
 
VLDB2015 会議報告
VLDB2015 会議報告VLDB2015 会議報告
VLDB2015 会議報告
 
AWS Redshift Analyzeの必要性とvacuumの落とし穴
AWS Redshift Analyzeの必要性とvacuumの落とし穴AWS Redshift Analyzeの必要性とvacuumの落とし穴
AWS Redshift Analyzeの必要性とvacuumの落とし穴
 
NAND Flash から InnoDB にかけての話(仮)
NAND Flash から InnoDB にかけての話(仮)NAND Flash から InnoDB にかけての話(仮)
NAND Flash から InnoDB にかけての話(仮)
 
40分でわかるHadoop徹底入門 (Cloudera World Tokyo 2014 講演資料)
40分でわかるHadoop徹底入門 (Cloudera World Tokyo 2014 講演資料) 40分でわかるHadoop徹底入門 (Cloudera World Tokyo 2014 講演資料)
40分でわかるHadoop徹底入門 (Cloudera World Tokyo 2014 講演資料)
 
Data Science Across Data Sources with Apache Arrow
Data Science Across Data Sources with Apache ArrowData Science Across Data Sources with Apache Arrow
Data Science Across Data Sources with Apache Arrow
 
Speed up UDFs with GPUs using the RAPIDS Accelerator
Speed up UDFs with GPUs using the RAPIDS AcceleratorSpeed up UDFs with GPUs using the RAPIDS Accelerator
Speed up UDFs with GPUs using the RAPIDS Accelerator
 
問合せ最適化インサイド
問合せ最適化インサイド問合せ最適化インサイド
問合せ最適化インサイド
 
Vacuum徹底解説
Vacuum徹底解説Vacuum徹底解説
Vacuum徹底解説
 
HIVE: Data Warehousing & Analytics on Hadoop
HIVE: Data Warehousing & Analytics on HadoopHIVE: Data Warehousing & Analytics on Hadoop
HIVE: Data Warehousing & Analytics on Hadoop
 

Semelhante a Amundsen: From discovering to security data

How Lyft Drives Data Discovery
How Lyft Drives Data DiscoveryHow Lyft Drives Data Discovery
How Lyft Drives Data DiscoveryNeo4j
 
Data Discovery and Metadata
Data Discovery and MetadataData Discovery and Metadata
Data Discovery and Metadatamarkgrover
 
Strata sf - Amundsen presentation
Strata sf - Amundsen presentationStrata sf - Amundsen presentation
Strata sf - Amundsen presentationTao Feng
 
Disrupting Data Discovery
Disrupting Data DiscoveryDisrupting Data Discovery
Disrupting Data Discoverymarkgrover
 
Data council sf amundsen presentation
Data council sf    amundsen presentationData council sf    amundsen presentation
Data council sf amundsen presentationTao Feng
 
Neo4j GraphTour Santa Monica 2019 - Amundsen Presentation
Neo4j GraphTour Santa Monica 2019 - Amundsen PresentationNeo4j GraphTour Santa Monica 2019 - Amundsen Presentation
Neo4j GraphTour Santa Monica 2019 - Amundsen PresentationTamikaTannis
 
How Lyft Drives Data Discovery
How Lyft Drives Data DiscoveryHow Lyft Drives Data Discovery
How Lyft Drives Data DiscoveryNeo4j
 
Advanced Analytics and Machine Learning with Data Virtualization
Advanced Analytics and Machine Learning with Data VirtualizationAdvanced Analytics and Machine Learning with Data Virtualization
Advanced Analytics and Machine Learning with Data VirtualizationDenodo
 
Solving Data Discovery Challenges at Lyft with Amundsen, an Open-source Metad...
Solving Data Discovery Challenges at Lyft with Amundsen, an Open-source Metad...Solving Data Discovery Challenges at Lyft with Amundsen, an Open-source Metad...
Solving Data Discovery Challenges at Lyft with Amundsen, an Open-source Metad...Databricks
 
Webinar: Lucidworks + Thomson Reuters for Improved Investment Performance
Webinar: Lucidworks + Thomson Reuters for Improved Investment PerformanceWebinar: Lucidworks + Thomson Reuters for Improved Investment Performance
Webinar: Lucidworks + Thomson Reuters for Improved Investment PerformanceLucidworks
 
Building a healthy data ecosystem around Kafka and Hadoop: Lessons learned at...
Building a healthy data ecosystem around Kafka and Hadoop: Lessons learned at...Building a healthy data ecosystem around Kafka and Hadoop: Lessons learned at...
Building a healthy data ecosystem around Kafka and Hadoop: Lessons learned at...Yael Garten
 
Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...
Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...
Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...Shirshanka Das
 
SDSC18 and DSATL Meetup March 2018
SDSC18 and DSATL Meetup March 2018 SDSC18 and DSATL Meetup March 2018
SDSC18 and DSATL Meetup March 2018 CareerBuilder.com
 
Democratizing Data within your organization - Data Discovery
Democratizing Data within your organization - Data DiscoveryDemocratizing Data within your organization - Data Discovery
Democratizing Data within your organization - Data DiscoveryMark Grover
 
The Great Lakes: How to Approach a Big Data Implementation
The Great Lakes: How to Approach a Big Data ImplementationThe Great Lakes: How to Approach a Big Data Implementation
The Great Lakes: How to Approach a Big Data ImplementationInside Analysis
 
Neo4j GraphDay Seattle- Sept19- Connected data imperative
Neo4j GraphDay Seattle- Sept19- Connected data imperativeNeo4j GraphDay Seattle- Sept19- Connected data imperative
Neo4j GraphDay Seattle- Sept19- Connected data imperativeNeo4j
 
Advanced Analytics and Machine Learning with Data Virtualization
Advanced Analytics and Machine Learning with Data VirtualizationAdvanced Analytics and Machine Learning with Data Virtualization
Advanced Analytics and Machine Learning with Data VirtualizationDenodo
 
Balancing Data Democracy with Data Privacy: The LinkedIn Story
Balancing Data Democracy with Data Privacy: The LinkedIn StoryBalancing Data Democracy with Data Privacy: The LinkedIn Story
Balancing Data Democracy with Data Privacy: The LinkedIn StoryAnthony Hsu
 
DataSpryng Overview
DataSpryng OverviewDataSpryng Overview
DataSpryng Overviewjkvr
 

Semelhante a Amundsen: From discovering to security data (20)

How Lyft Drives Data Discovery
How Lyft Drives Data DiscoveryHow Lyft Drives Data Discovery
How Lyft Drives Data Discovery
 
Data Discovery and Metadata
Data Discovery and MetadataData Discovery and Metadata
Data Discovery and Metadata
 
Meetup SF - Amundsen
Meetup SF  -  AmundsenMeetup SF  -  Amundsen
Meetup SF - Amundsen
 
Strata sf - Amundsen presentation
Strata sf - Amundsen presentationStrata sf - Amundsen presentation
Strata sf - Amundsen presentation
 
Disrupting Data Discovery
Disrupting Data DiscoveryDisrupting Data Discovery
Disrupting Data Discovery
 
Data council sf amundsen presentation
Data council sf    amundsen presentationData council sf    amundsen presentation
Data council sf amundsen presentation
 
Neo4j GraphTour Santa Monica 2019 - Amundsen Presentation
Neo4j GraphTour Santa Monica 2019 - Amundsen PresentationNeo4j GraphTour Santa Monica 2019 - Amundsen Presentation
Neo4j GraphTour Santa Monica 2019 - Amundsen Presentation
 
How Lyft Drives Data Discovery
How Lyft Drives Data DiscoveryHow Lyft Drives Data Discovery
How Lyft Drives Data Discovery
 
Advanced Analytics and Machine Learning with Data Virtualization
Advanced Analytics and Machine Learning with Data VirtualizationAdvanced Analytics and Machine Learning with Data Virtualization
Advanced Analytics and Machine Learning with Data Virtualization
 
Solving Data Discovery Challenges at Lyft with Amundsen, an Open-source Metad...
Solving Data Discovery Challenges at Lyft with Amundsen, an Open-source Metad...Solving Data Discovery Challenges at Lyft with Amundsen, an Open-source Metad...
Solving Data Discovery Challenges at Lyft with Amundsen, an Open-source Metad...
 
Webinar: Lucidworks + Thomson Reuters for Improved Investment Performance
Webinar: Lucidworks + Thomson Reuters for Improved Investment PerformanceWebinar: Lucidworks + Thomson Reuters for Improved Investment Performance
Webinar: Lucidworks + Thomson Reuters for Improved Investment Performance
 
Building a healthy data ecosystem around Kafka and Hadoop: Lessons learned at...
Building a healthy data ecosystem around Kafka and Hadoop: Lessons learned at...Building a healthy data ecosystem around Kafka and Hadoop: Lessons learned at...
Building a healthy data ecosystem around Kafka and Hadoop: Lessons learned at...
 
Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...
Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...
Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...
 
SDSC18 and DSATL Meetup March 2018
SDSC18 and DSATL Meetup March 2018 SDSC18 and DSATL Meetup March 2018
SDSC18 and DSATL Meetup March 2018
 
Democratizing Data within your organization - Data Discovery
Democratizing Data within your organization - Data DiscoveryDemocratizing Data within your organization - Data Discovery
Democratizing Data within your organization - Data Discovery
 
The Great Lakes: How to Approach a Big Data Implementation
The Great Lakes: How to Approach a Big Data ImplementationThe Great Lakes: How to Approach a Big Data Implementation
The Great Lakes: How to Approach a Big Data Implementation
 
Neo4j GraphDay Seattle- Sept19- Connected data imperative
Neo4j GraphDay Seattle- Sept19- Connected data imperativeNeo4j GraphDay Seattle- Sept19- Connected data imperative
Neo4j GraphDay Seattle- Sept19- Connected data imperative
 
Advanced Analytics and Machine Learning with Data Virtualization
Advanced Analytics and Machine Learning with Data VirtualizationAdvanced Analytics and Machine Learning with Data Virtualization
Advanced Analytics and Machine Learning with Data Virtualization
 
Balancing Data Democracy with Data Privacy: The LinkedIn Story
Balancing Data Democracy with Data Privacy: The LinkedIn StoryBalancing Data Democracy with Data Privacy: The LinkedIn Story
Balancing Data Democracy with Data Privacy: The LinkedIn Story
 
DataSpryng Overview
DataSpryng OverviewDataSpryng Overview
DataSpryng Overview
 

Mais de markgrover

From discovering to trusting data
From discovering to trusting dataFrom discovering to trusting data
From discovering to trusting datamarkgrover
 
Amundsen lineage designs - community meeting, Dec 2020
Amundsen lineage designs - community meeting, Dec 2020 Amundsen lineage designs - community meeting, Dec 2020
Amundsen lineage designs - community meeting, Dec 2020 markgrover
 
Amundsen at Brex and Looker integration
Amundsen at Brex and Looker integrationAmundsen at Brex and Looker integration
Amundsen at Brex and Looker integrationmarkgrover
 
REA Group's journey with Data Cataloging and Amundsen
REA Group's journey with Data Cataloging and AmundsenREA Group's journey with Data Cataloging and Amundsen
REA Group's journey with Data Cataloging and Amundsenmarkgrover
 
Amundsen gremlin proxy design
Amundsen gremlin proxy designAmundsen gremlin proxy design
Amundsen gremlin proxy designmarkgrover
 
Amundsen: From discovering to security data
Amundsen: From discovering to security dataAmundsen: From discovering to security data
Amundsen: From discovering to security datamarkgrover
 
Data Discovery & Trust through Metadata
Data Discovery & Trust through MetadataData Discovery & Trust through Metadata
Data Discovery & Trust through Metadatamarkgrover
 
The Lyft data platform: Now and in the future
The Lyft data platform: Now and in the futureThe Lyft data platform: Now and in the future
The Lyft data platform: Now and in the futuremarkgrover
 
TensorFlow Extension (TFX) and Apache Beam
TensorFlow Extension (TFX) and Apache BeamTensorFlow Extension (TFX) and Apache Beam
TensorFlow Extension (TFX) and Apache Beammarkgrover
 
Big Data at Speed
Big Data at SpeedBig Data at Speed
Big Data at Speedmarkgrover
 
Near real-time anomaly detection at Lyft
Near real-time anomaly detection at LyftNear real-time anomaly detection at Lyft
Near real-time anomaly detection at Lyftmarkgrover
 
Dogfooding data at Lyft
Dogfooding data at LyftDogfooding data at Lyft
Dogfooding data at Lyftmarkgrover
 
Fighting cybersecurity threats with Apache Spot
Fighting cybersecurity threats with Apache SpotFighting cybersecurity threats with Apache Spot
Fighting cybersecurity threats with Apache Spotmarkgrover
 
Fraud Detection with Hadoop
Fraud Detection with HadoopFraud Detection with Hadoop
Fraud Detection with Hadoopmarkgrover
 
Top 5 mistakes when writing Spark applications
Top 5 mistakes when writing Spark applicationsTop 5 mistakes when writing Spark applications
Top 5 mistakes when writing Spark applicationsmarkgrover
 
Top 5 mistakes when writing Spark applications
Top 5 mistakes when writing Spark applicationsTop 5 mistakes when writing Spark applications
Top 5 mistakes when writing Spark applicationsmarkgrover
 
Architecting Applications with Hadoop
Architecting Applications with HadoopArchitecting Applications with Hadoop
Architecting Applications with Hadoopmarkgrover
 
SQL Engines for Hadoop - The case for Impala
SQL Engines for Hadoop - The case for ImpalaSQL Engines for Hadoop - The case for Impala
SQL Engines for Hadoop - The case for Impalamarkgrover
 
Intro to hadoop tutorial
Intro to hadoop tutorialIntro to hadoop tutorial
Intro to hadoop tutorialmarkgrover
 
NYC HUG - Application Architectures with Apache Hadoop
NYC HUG - Application Architectures with Apache HadoopNYC HUG - Application Architectures with Apache Hadoop
NYC HUG - Application Architectures with Apache Hadoopmarkgrover
 

Mais de markgrover (20)

From discovering to trusting data
From discovering to trusting dataFrom discovering to trusting data
From discovering to trusting data
 
Amundsen lineage designs - community meeting, Dec 2020
Amundsen lineage designs - community meeting, Dec 2020 Amundsen lineage designs - community meeting, Dec 2020
Amundsen lineage designs - community meeting, Dec 2020
 
Amundsen at Brex and Looker integration
Amundsen at Brex and Looker integrationAmundsen at Brex and Looker integration
Amundsen at Brex and Looker integration
 
REA Group's journey with Data Cataloging and Amundsen
REA Group's journey with Data Cataloging and AmundsenREA Group's journey with Data Cataloging and Amundsen
REA Group's journey with Data Cataloging and Amundsen
 
Amundsen gremlin proxy design
Amundsen gremlin proxy designAmundsen gremlin proxy design
Amundsen gremlin proxy design
 
Amundsen: From discovering to security data
Amundsen: From discovering to security dataAmundsen: From discovering to security data
Amundsen: From discovering to security data
 
Data Discovery & Trust through Metadata
Data Discovery & Trust through MetadataData Discovery & Trust through Metadata
Data Discovery & Trust through Metadata
 
The Lyft data platform: Now and in the future
The Lyft data platform: Now and in the futureThe Lyft data platform: Now and in the future
The Lyft data platform: Now and in the future
 
TensorFlow Extension (TFX) and Apache Beam
TensorFlow Extension (TFX) and Apache BeamTensorFlow Extension (TFX) and Apache Beam
TensorFlow Extension (TFX) and Apache Beam
 
Big Data at Speed
Big Data at SpeedBig Data at Speed
Big Data at Speed
 
Near real-time anomaly detection at Lyft
Near real-time anomaly detection at LyftNear real-time anomaly detection at Lyft
Near real-time anomaly detection at Lyft
 
Dogfooding data at Lyft
Dogfooding data at LyftDogfooding data at Lyft
Dogfooding data at Lyft
 
Fighting cybersecurity threats with Apache Spot
Fighting cybersecurity threats with Apache SpotFighting cybersecurity threats with Apache Spot
Fighting cybersecurity threats with Apache Spot
 
Fraud Detection with Hadoop
Fraud Detection with HadoopFraud Detection with Hadoop
Fraud Detection with Hadoop
 
Top 5 mistakes when writing Spark applications
Top 5 mistakes when writing Spark applicationsTop 5 mistakes when writing Spark applications
Top 5 mistakes when writing Spark applications
 
Top 5 mistakes when writing Spark applications
Top 5 mistakes when writing Spark applicationsTop 5 mistakes when writing Spark applications
Top 5 mistakes when writing Spark applications
 
Architecting Applications with Hadoop
Architecting Applications with HadoopArchitecting Applications with Hadoop
Architecting Applications with Hadoop
 
SQL Engines for Hadoop - The case for Impala
SQL Engines for Hadoop - The case for ImpalaSQL Engines for Hadoop - The case for Impala
SQL Engines for Hadoop - The case for Impala
 
Intro to hadoop tutorial
Intro to hadoop tutorialIntro to hadoop tutorial
Intro to hadoop tutorial
 
NYC HUG - Application Architectures with Apache Hadoop
NYC HUG - Application Architectures with Apache HadoopNYC HUG - Application Architectures with Apache Hadoop
NYC HUG - Application Architectures with Apache Hadoop
 

Último

How to Build an Experimentation Culture for Data-Driven Product Development
How to Build an Experimentation Culture for Data-Driven Product DevelopmentHow to Build an Experimentation Culture for Data-Driven Product Development
How to Build an Experimentation Culture for Data-Driven Product DevelopmentAggregage
 
Air Con Energy Rating Info411 Presentation.pdf
Air Con Energy Rating Info411 Presentation.pdfAir Con Energy Rating Info411 Presentation.pdf
Air Con Energy Rating Info411 Presentation.pdfJasonBoboKyaw
 
Stochastic Dynamic Programming and You.pptx
Stochastic Dynamic Programming and You.pptxStochastic Dynamic Programming and You.pptx
Stochastic Dynamic Programming and You.pptxjkmrshll88
 
Bengaluru Tableau UG event- 2nd March 2024 Q1
Bengaluru Tableau UG event- 2nd March 2024 Q1Bengaluru Tableau UG event- 2nd March 2024 Q1
Bengaluru Tableau UG event- 2nd March 2024 Q1bengalurutug
 
Understanding the Impact of video length on student performance
Understanding the Impact of video length on student performanceUnderstanding the Impact of video length on student performance
Understanding the Impact of video length on student performancePrithaVashisht1
 
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdfNeo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdfNeo4j
 
Báo cáo Social Media Benchmark 2024 cho dân Marketing
Báo cáo Social Media Benchmark 2024 cho dân MarketingBáo cáo Social Media Benchmark 2024 cho dân Marketing
Báo cáo Social Media Benchmark 2024 cho dân MarketingMarketingTrips
 
Empowering Decisions A Guide to Embedded Analytics
Empowering Decisions A Guide to Embedded AnalyticsEmpowering Decisions A Guide to Embedded Analytics
Empowering Decisions A Guide to Embedded AnalyticsGain Insights
 
Prediction Of Cryptocurrency Prices Using Lstm, Svm And Polynomial Regression...
Prediction Of Cryptocurrency Prices Using Lstm, Svm And Polynomial Regression...Prediction Of Cryptocurrency Prices Using Lstm, Svm And Polynomial Regression...
Prediction Of Cryptocurrency Prices Using Lstm, Svm And Polynomial Regression...ferisulianta.com
 
Data Analytics Fundamentals: data analytics types.potx
Data Analytics Fundamentals: data analytics types.potxData Analytics Fundamentals: data analytics types.potx
Data Analytics Fundamentals: data analytics types.potxEmmanuel Dauda
 
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptxSTOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptxFurkanTasci3
 
The market for cross-border mortgages in Europe
The market for cross-border mortgages in EuropeThe market for cross-border mortgages in Europe
The market for cross-border mortgages in Europe321k
 
Data Collection from Social Media Platforms
Data Collection from Social Media PlatformsData Collection from Social Media Platforms
Data Collection from Social Media PlatformsMahmoud Yasser
 
TCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTimothy Spann
 
Enabling GenAI Breakthroughs with Knowledge Graphs
Enabling GenAI Breakthroughs with Knowledge GraphsEnabling GenAI Breakthroughs with Knowledge Graphs
Enabling GenAI Breakthroughs with Knowledge GraphsNeo4j
 
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptxSTOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptxFurkanTasci3
 
Paul Martin (Gartner) - Show Me the AI Money.pdf
Paul Martin (Gartner) - Show Me the AI Money.pdfPaul Martin (Gartner) - Show Me the AI Money.pdf
Paul Martin (Gartner) - Show Me the AI Money.pdfdcphostmaster
 
Microeconomic Group Presentation Apple.pdf
Microeconomic Group Presentation Apple.pdfMicroeconomic Group Presentation Apple.pdf
Microeconomic Group Presentation Apple.pdfmxlos0
 
2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-Profits2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-ProfitsTimothy Spann
 
PPT for Presiding Officer.pptxvvdffdfgggg
PPT for Presiding Officer.pptxvvdffdfggggPPT for Presiding Officer.pptxvvdffdfgggg
PPT for Presiding Officer.pptxvvdffdfggggbhadratanusenapati1
 

Último (20)

How to Build an Experimentation Culture for Data-Driven Product Development
How to Build an Experimentation Culture for Data-Driven Product DevelopmentHow to Build an Experimentation Culture for Data-Driven Product Development
How to Build an Experimentation Culture for Data-Driven Product Development
 
Air Con Energy Rating Info411 Presentation.pdf
Air Con Energy Rating Info411 Presentation.pdfAir Con Energy Rating Info411 Presentation.pdf
Air Con Energy Rating Info411 Presentation.pdf
 
Stochastic Dynamic Programming and You.pptx
Stochastic Dynamic Programming and You.pptxStochastic Dynamic Programming and You.pptx
Stochastic Dynamic Programming and You.pptx
 
Bengaluru Tableau UG event- 2nd March 2024 Q1
Bengaluru Tableau UG event- 2nd March 2024 Q1Bengaluru Tableau UG event- 2nd March 2024 Q1
Bengaluru Tableau UG event- 2nd March 2024 Q1
 
Understanding the Impact of video length on student performance
Understanding the Impact of video length on student performanceUnderstanding the Impact of video length on student performance
Understanding the Impact of video length on student performance
 
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdfNeo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
 
Báo cáo Social Media Benchmark 2024 cho dân Marketing
Báo cáo Social Media Benchmark 2024 cho dân MarketingBáo cáo Social Media Benchmark 2024 cho dân Marketing
Báo cáo Social Media Benchmark 2024 cho dân Marketing
 
Empowering Decisions A Guide to Embedded Analytics
Empowering Decisions A Guide to Embedded AnalyticsEmpowering Decisions A Guide to Embedded Analytics
Empowering Decisions A Guide to Embedded Analytics
 
Prediction Of Cryptocurrency Prices Using Lstm, Svm And Polynomial Regression...
Prediction Of Cryptocurrency Prices Using Lstm, Svm And Polynomial Regression...Prediction Of Cryptocurrency Prices Using Lstm, Svm And Polynomial Regression...
Prediction Of Cryptocurrency Prices Using Lstm, Svm And Polynomial Regression...
 
Data Analytics Fundamentals: data analytics types.potx
Data Analytics Fundamentals: data analytics types.potxData Analytics Fundamentals: data analytics types.potx
Data Analytics Fundamentals: data analytics types.potx
 
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptxSTOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
 
The market for cross-border mortgages in Europe
The market for cross-border mortgages in EuropeThe market for cross-border mortgages in Europe
The market for cross-border mortgages in Europe
 
Data Collection from Social Media Platforms
Data Collection from Social Media PlatformsData Collection from Social Media Platforms
Data Collection from Social Media Platforms
 
TCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI Pipelines
 
Enabling GenAI Breakthroughs with Knowledge Graphs
Enabling GenAI Breakthroughs with Knowledge GraphsEnabling GenAI Breakthroughs with Knowledge Graphs
Enabling GenAI Breakthroughs with Knowledge Graphs
 
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptxSTOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
 
Paul Martin (Gartner) - Show Me the AI Money.pdf
Paul Martin (Gartner) - Show Me the AI Money.pdfPaul Martin (Gartner) - Show Me the AI Money.pdf
Paul Martin (Gartner) - Show Me the AI Money.pdf
 
Microeconomic Group Presentation Apple.pdf
Microeconomic Group Presentation Apple.pdfMicroeconomic Group Presentation Apple.pdf
Microeconomic Group Presentation Apple.pdf
 
2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-Profits2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-Profits
 
PPT for Presiding Officer.pptxvvdffdfgggg
PPT for Presiding Officer.pptxvvdffdfggggPPT for Presiding Officer.pptxvvdffdfgggg
PPT for Presiding Officer.pptxvvdffdfgggg
 

Amundsen: From discovering to security data

  • 1. Mark Grover | Lyft | @mark_grover Alyssa Ransbury | Square | @alyssaran Slides: cutt.ly/a6n Amundsen: From Discovering to Securing Data
  • 2. Who we are • Product Manager at Lyft • Committer/PMC on various Apache projects • Previously developer on Spark at Cloudera 2 • Security Engineer at Square • Leads and supports on privacy engineering efforts across dozens of product teams
  • 4. >35% consumer time spent on discovering & validating trustworthy data 4 Analyst/DS workflow and time spent on each step Other side effects 1. Interrupt heavy culture ● What data should I use for X? ● Is that trustworthy? 2. Increased DB load SELECT * … LIMIT 100 SELECT COUNT(*) over x days
  • 6. 1. What kind of information? (aka ABC of metadata) 6 Application Context Metadata needed by humans or applications to operate ● Where is the data? ● What are the semantics of the data? Behavior How is data created and used over time? ● Who’s using the data? ● Who created the data? Change Change in data over time ● How is the data evolving over time? ● Evolution of code that generates the data Terminology borrowed from Ground paper Data Lineage TODAY
  • 7. Short answer: Any data within your organization Long answer: 2. About what data? 7 Data stores Schema registry Events / Schemas StreamsPeople Employees TODAY NotebooksDashboard / Reports Processes
  • 8. Develop breadth of applications 8 Metadata Data Discovery & Trust Compliance (GDPR / CCPA / Financial) Security / ACL Data Monitoring / Ops Data Quality Cost Mgmt Data Mainten- ance
  • 9. Search based Lineage based Network based Programmatic Where is the table/dashboard for X? What does it contain? I am changing a data model, who are the owner and most common users? I want to follow a power user in my team. Access metadata programmatically Does this analysis already exist? This table’s delivery was delayed today, I want to notify everyone downstream. I want to bookmark tables of interest and get a feed of data delay, schema change, incidents. Put (pull / push) metadata programmatically Other requirements ● Leverage as much data automatically as possible ● Preferably, open source and healthy community ● Easy to set up Goal: Reduce time to find trusted data w/ versatile graph
  • 10. Meet Amundsen 10 First person to discover the South Pole - Norwegian explorer, Roald Amundsen
  • 11. Search for data, or browse
  • 13. See details of the data set
  • 14. See detailed descriptions and profile of the column
  • 15. Preview existing data, if perms allow
  • 16. Search for existing dashboards (aka reports) 16
  • 19. Search for data owned and used by your peers!
  • 21. 21 “This is God’s work” - Head of Analytics, Lyft “I was on call and I’m confident 50% of the questions could have been answered by a simple search in Amundsen” - Data Scientist,Lyft A6n @ Lyft: 750+ WAUs, 150k+ tables, 4k+ employee pages
  • 22. Roles of Amundsen users at Lyft 22 Penetration rate: DS (aka analyst): 81% RS (aka DS): 71% PM: 22% SWE: 17% Cust Serv: 7% (12/390) Sp. Ops: 67% (10/15) Sp. Op Leads: 53% (9/17) Economist: 100% (7/7) Cust. Quality: 78% (7/9) Growth Mktg: 25% (6/24)
  • 24. Roadmap (subject to change, not ordered) • Tighter Lineage integration / visualization • Better view integration • ACL integration, allow only specific roles to edit descriptions • Show search context for what matched • Index more resources (notebooks, Kafka topics, etc.) 24
  • 25. Amundsen Open Source 25 700 Community members 150+ Companies in the community 20+ Companies using in production
  • 26. Amundsen Open Source Community 26 ProminentusersActivecommunity
  • 27. Develop breadth of applications 27 Metadata Data Discovery & Trust Compliance (GDPR / CCPA / Financial) Security / ACL Data Monitoring / Ops Data Quality Cost Mgmt Data Mainten- ance
  • 29. 29 Postgres Hive Redshift ... Presto Github Source File Databuilder Crawler Neo4j Elastic Search Metadata Service Search Service Frontend ServiceML Feature Service Security Service Other Microservices Metadata Sources
  • 30. 30 Postgres Hive Redshift ... Presto Github Source File Databuilder Crawler Neo4j Elastic Search Metadata Service Search Service Frontend ServiceML Feature Service Security Service Other Microservices Metadata Sources
  • 31. 31 Postgres Hive Redshift ... Presto Github Source File Databuilder Crawler Neo4j Elastic Search Metadata Service Search Service Frontend ServiceML Feature Service Security Service Other Microservices Metadata Sources
  • 32. 32 Postgres Hive Redshift ... Presto Github Source File Databuilder Crawler Neo4j Elastic Search Metadata Service Search Service Frontend ServiceML Feature Service Security Service Other Microservices Metadata Sources
  • 33. 33 Postgres Hive Redshift ... Presto Github Source File Databuilder Crawler Neo4j Elastic Search Metadata Service Search Service Frontend ServiceML Feature Service Security Service Other Microservices Metadata Sources
  • 35. Benefits of a graph
  • 36. The Problem 36 ● Relied mostly on manual work to understand our data ● We have A LOT of data, and growing ● NOT scalable ● Lots of goals related to data privacy on top of requirements from laws like GDPR / CCPA
  • 37. 37 Postgres Hive Redshift ... Presto Github Source File Databuilder Crawler Neo4j Elastic Search Metadata Service Search Service Frontend ServiceML Feature Service Security Service Other Microservices Metadata Sources 1. Store more specific metadata 2. Automate data inventorying 3. Support more granular access controls
  • 38. 1. Store more specific metadata 38
  • 39. 39 Postgres Hive Redshift ... Presto Github Source File Databuilder Crawler Neo4j Elastic Search Metadata Service Search Service Frontend ServiceML Feature Service Security Service Other Microservices Metadata Sources 1. Store more specific metadata
  • 41. PII Semantic Type for data in Column PII Semantic Type 41 A set value describing the general contents of data in a column, with a focus on discovering data that fits into one of three buckets ● Sensitive by itself ● Could be sensitive when taken together with other data ● Links to sensitive data, like an internal identification token Some Column Data
  • 42. Data Storage Security 42 A set value describing the form of data in a column PII Semantic Type for data in Column Some Column Data Data Storage Security for data in Column
  • 43. Data Subject Type 43 A set value describing users whose information exists in a column ● Example: square_product_user PII Semantic Type for data in Column Some Column Data Data Storage Security for data in Column Data Subject Type for data in Column
  • 47. 47 Postgres Hive Redshift ... Presto Github Source File Databuilder Crawler Neo4j Elastic Search Metadata Service Search Service Frontend ServiceML Feature Service Security Service Other Microservices Metadata Sources 2. Automate data inventorying
  • 48. Why? ● We have a lot of data! ● Metadata should be kept current at scale ● New data sources get created all the time ● Mistakes happen 48
  • 49. Flagging Possibly Sensitive Data Locations 49 Some Column Data Data Storage Security encrypted
  • 50. Flagging Possibly Sensitive Data Locations 50 Some Column Data Data Storage Security encrypted Flag
  • 51. Flags Contain information about: ● The metadata type it relates to (i.e. PII Semantic Type, Data Storage Security) ● The possible value (i.e. encrypted for Data Storage Security) ● Human readable description 51 Some Column Data Flag Reason Result Reviewer
  • 52. Checking for Data Sensitivity ● Column name ○ Keyword partial or full match ○ Literal PII match ● Column type ● Table name ● Sample values + Google DLP API 52
  • 54. User Flow 54 Some Column Data Flag Data Storage Security plaintext Result: True Reviewer: person 1 Data Storage Security plaintext Flag PII Semantic Type phone number Result: False Reviewer: person 2
  • 57. 3. Support more granular access controls 57
  • 58. 58 Postgres Hive Redshift ... Presto Github Source File Databuilder Crawler Neo4j Elastic Search Metadata Service Search Service Frontend ServiceML Feature Service Security Service Other Microservices Metadata Sources 3. Support more granular access controls
  • 59. Capability-governed access control ● Purpose - An enumerated, legal-approved reason to access specific PII Semantic Types ● Capability - A combination of Purpose and Data Subject Type 59
  • 61. Capability-governed access control 61 Capability 001 USER SUPPORT CASH APP CUSTOMER PERSON NAME EMAIL ADDRESS
  • 64. Future and Work in Progress ● Gremlin Proxy ● UI components ○ show users exactly the data they have access to based on their capabilities ○ show the data that is tied to a specific capability ○ empower data owners to revoke broad access to some data ● Supporting additional data types ● And beyond! (we are hiring…) 64
  • 66. Develop breadth of applications 66 Metadata Data Discovery & Trust Compliance (GDPR / CCPA / Financial) Security / ACL Data Monitoring / Ops Data Quality Cost Mgmt Data Mainten- ance
  • 67. Get started! Check out Github: github.com/lyft/amundsen Join Slack (~700 users): Link to join on github 67 Check out other blogs and videos: Amundsen for Privacy @ Square Introducing Amundsen Amundsen's architecture YouTube channel
  • 68. Thanks! 68 Mark Grover | Lyft | @mark_grover Alyssa Ransbury | Square | @alyssaran Icons under Creative Commons License from https://thenounproject.com/