Data Mesh for Dinner

4. © 2021 Snowflake Inc. All Rights Reserved Inventor Thoughtworks: >8000 employees High-end Consulting Customers such as Daimler, Delta, Otto… DATA MESH IS ONLY MENTIONED IN RELATION TO SNOWFLAKE AND DELTA LAKE

5. © 2021 Snowflake Inc. All Rights Reserved Data Mesh https://martinfowler.com/articles/data-monolith-to-mesh.html • Originated at Thoughtworks in 2019/2020 • Driven by Zhamak Dehghani • Data Mesh Community: https://datameshlearning.com/ https://martinfowler.com/articles/data-mesh-principles.html

6. Motivations for a Data Mesh Limitations of centralized warehouses and lakes Diverse data sources from many domains Diverse set of consumers and requirements Monolithic DWH or Data Lake ETL ETL ETL ELT ELT ELT Data Model • Proliferation of complex ETL / ELT processes • Centralized data engineering team lacks data source domain knowledge • Limited ability to address data quality issues at the sources or react to change • Difficult to scale to many sources & consumers, slow to integrate new sources ETL/ELT pipelines ETL ETL ETL ETL ETL ELT ELT Data Model Data Model Data Model © 2021 Snowflake Inc. All Rights Reserved

7. © 2021 Snowflake Inc. All Rights Reserved DATA MESH PRINCIPLES Distribute responsibility for data pipelines and data quality to people with domain knowledge. Serve data as-a-product using a common self-service IT infrastructure platform. Domain-Centric Ownership & Architecture Data as-a-Product Self-Serve Infrastructure as-a- Platform Federated Governance ● Data is discoverable ● Data is easy to obtain and use ● Data is documented ● Domains responsible for the quality of their data ● Domain-agnostic, common tool set ● Easy to use and low maintenance to support ● Easy to deploy repeatable patterns for common requirements: cleansing, transformation, automation, storage, security, governance, sharing ● Global interoperability standards across domains ● Define and use global data governance policies ● Define and apply governance within each domain and propagate downstream ● Data pipelines owned by teams with domain knowledge ● Domains own cleansing, refinement, historization, pre- aggregation, etc. ● Domains responsible for governance, lineage, etc. ● Domains treat data with consumers in mind

8. Data Mesh: Domain-centric Architecture Data Domain 1 Data sources from different domains Consumers • Domain-centric ownership of data sources, pipelines, and data quality • Ownership sits with domain knowledge --> better data quality for consumers • Domain teams can react faster to source format changes or quality issues • Overall easier to scale the number of sources & consumers • Consumers pull from >= 1 domains • Data assets offered as products • “Serve & pull” instead of “push & ingest” model Data Domain 2 Data Domain 4 Interoperability Standards, Federated Governance, Data Catalog Data Domain 5 ELT ELT ETL ETL ETL Data Model Data Model ETL ETL ETL ETL ETL ETL ETL Data Domain 3 Data Domain 6

9. © 2021 Snowflake Inc. All Rights Reserved Data Mesh changes Scope and Responsibilities Different split of responsibilities! One team per data product domain, responsible across all or most stages of the data life cycle. Separate teams for different stages of bringing data from sources to consumers. Data Mesh is a “socio-technical shift”. An organizational paradigm for managing, sharing, and democratizing data.

10. © 2020 Snowflake Inc. All Rights Reserved Data Domains vs Data Silos Data Domains Data Silos Treat data as a product with consumers in mind Yes No Data easily discoverable Yes No Data is documented for consumers Yes No Downstream consumers have easy access Yes No Global data interoperability standards Yes No Responsible for data quality for the benefit of downstream consumers Yes No Ownership of data pipelines to expose data in forms and shapes that are useful for others Yes No Historical snapshots and aggregations for the benefit of data consumers Yes No Common self-service infrastructure Yes No Federated governance Yes No

12. © 2021 Snowflake Inc. All Rights Reserved BENEFITS OF SNOWFLAKE FOR A DATA MESH Snowflake extends a Data Mesh approach by enabling domains to not only share data as a product but also processing logic as a product Domain-Centric Ownership & Architecture Data as-a-Product Self-Serve Infrastructure as-a- Platform Federated Governance ● Enables sharing of data and functions as products (public or private) via secure data sharing ● Out-of-the box capabilities to publish, discover, request, and access data products ● Data monetization available for new revenue streams and easier procurement ● Fully managed for ease of use and near-zero maintenance ● Instantly deploy or scale compute resources ● Rich set of capabilities across workloads ● Native governance controls that follow the data across accounts and clouds through Snowgrid ● Data Marketplace offers cataloguing for data products ● Data exchange allows private governed sharing ● Integrated with leading governance tools ● Monitoring and alerting on compute ● Designed for distributed use in a global cloud network (Snowgrid) ● Separate compute clusters ● Data sharing within Data Cloud ● Easy to maintain cost governance within and across domains ● Can have as many databases and accounts as you want

13. © 2021 Snowflake Inc. All Rights Reserved THE DATA CLOUD IS A GLOBAL DATA MESH! A software company shares terabytes of data with hundreds of customers COVID-19 data is available live on Snowflake Data Marketplace from a US State, and other organizations Today’s financial data is accessible immediately without data pipelines Thousands of companies share data with suppliers, partners, or other business units * Visualization based on actual Data Cloud sharing activity as of Oct 28 2021 THOUSANDS OF ORGANIZATIONS ARE SHARING DATA WITH THEIR ECOSYSTEM

14. Data Mesh Reference Architecture Domain: Customer Domain: Sales Domain: Products Domain: Marketing Domain: Customer 360 Inventory of shared data products Snowflake Reader Account Snowflake Data Cloud Consumers Data Sources Interoperability Standards, Federated Governance, 3rd Party Tools Snowflake Data Sharing as the preferred interoperability standard. Data Marketplace makes data discoverable. Data Marketplace / Catalog: • Connects providers to consumers • Inventory of available assets • No central storage of shared data • Providers retain full control over shared assets (data, functions) • Consumers access live provider data, no copies or ETL required Data domains: • Can consume and share data or functions • Control access policies, data masking, etc. for downstream consumers • Can share external tables, i.e. provide access to data outside of Snowflake • Can provide reader accounts for non-Snowflake consumers Data Consumers: • Register shared data for local SQL access in their environment (no copy) Snowflake Data Marketplace or 3rd-party catalog 3rd party marketing agency Reseller Sales Analysts Churn & Retention Business optimization Finance & Controlling

15. © 2021 Snowflake Inc. All Rights Reserved Snowflake Data Marketplace Contact & Support Sample SQL Data product descriptions, meta data Searchable inventory of data products Request & access process

16. Global and Multi-Cloud Data Mesh Data Domain 1 Data Domain 2 Data Domain 3 Data Domain 5 Data Domain 4 Interoperability Standards, Federated Governance, 3rd Party Tools US East FRA Snowflake Reader Account Consumers Snowflake enables a truly global and multi-cloud data mesh across cloud platforms and regions. • Data sources, data domains, and consumers can sit in different regions and different cloud platforms • Snowflake enables a truly global and multi-cloud data mesh Tokyo Zurich Data Sources Snowflake Data Cloud Inventory of shared data products Snowflake Data Market- place or 3rd- party catalog

17. Snowflake Data Cloud Data Sources Interoperability Standards, Federated Governance, 3rd Party Tools Data Mesh Reference Architecture: Governance Data Domain 1 Data Domain 2 Data Domain 3 Data Domain 5 Data Domain 4 Snowflake Reader Account Consumers In the Snowflake Data Cloud the governance follows the data ! 1 2 3 4 Create Listing Create local link Creat e local link 5 6 Data Mesh with Federated Governance: • Domain 2 shares a data asset by creating a listing in the marketplace • Domain 2 also defines access restrictions on that data asset, based on consumer roles and other attributes • Domain 5 and consumer 3 discover the asset in the inventory and link to it in their local environment for live SQL access against the table in domain 2 • Domain 5 and consumer 3 are subject to the access restrictions defined by domain 2 • Domain 5 transforms the data, defines additional access restrictions , and shares a copy with consumer 6 • Consumer 6 can use the data, subject to the propagated access restrictions that domain 5 has added. Inventory of shared data as a product assets via Data Marketplace or third-party tools

18. © 2020 Snowflake Inc. All Rights Reserved Important Data Mesh Considerations Data domains must not be data silos Data Mesh requires organizational changes Consistent use of federated governance How to scale the number of data domains? Data-as-a-product is a culture shift Data Mesh changes responsibilities and processes Consistent use of meta data, lineage, and data quality metrics Cost control across distributed domains? Which form of centralized control? Common self-service tool set for all domains Consistent use of interoperability standards Avoid duplication of effort across domains

19. © 2021 Snowflake Inc. All Rights Reserved Data Mesh Summary • Data Mesh: benefits and challenges for enterprise data management • Snowflake offers unique benefits for data mesh implementations • Snowflake’s distributed cross-cloud platform is a natural fit for distributed domains • Ease of use and near-zero maintenance: true self-service for a broad range of data pipeline and data management capabilities • Data Marketplace and Data Sharing: Out-of-the box capabilities to publish, discover, request, and obtain data products • Live access to data products across domains, no ETL or copies required • Data as a product as well as function as a product • Native governance controls follow the data across domains • Easy integration with leading 3rd party tools Domain-centric Ownership and Architecture Data as a Product Self-Serve Infrastructure as a Platform Federated Governance

22. © 2021 Snowflake Inc. All Rights Reserved Replication Global Data Marketplace Reader Account Direct data sharing Offer data or functions Consume data or functions Region Replication Consume data/functions Repl. Failover / Failback 2-way sharing • Currently 24 cloud regions • Integrated into a single cloud data network • Eliminates data silos • Enables secure sharing of live data & functions, no ETL • Enables replication for DR and failover • Enables global Data Mesh architectures Private Data Marketplace Failover / Failback Snowgrid: the backbone of a global data mesh Sample set of Snowflake nodes exploiting Snowgrid capabilities

23. © 2021 Snowflake Inc. All Rights Reserved Secure Data Sharing Within and Across Regions Provider SHARE Consumer A Enriched Table View, Query, Join SHARE Consumer B Provider Replica SHARE Consumer D Consumer C Database Replication Share Replication* Data Cleanroom S3 Database Replication Share Replication* Consumer

24. SOURCES Zhamak's initial article introducing the data mesh concept: https://martinfowler.com/articles/data-monolith-to-mesh.html Sven Balnojan's article on how one could implement a data mesh: https://towardsdatascience.com/data-mesh-applied- 21bed87876f2 Zhamak's second article, delving deeper: https://martinfowler.com/articles/data-mesh-principles.html Barr Moses' article on Data Mesh basics: https://towardsdatascience.com/what-is-a-data-mesh-and-how-not-to-mesh-it-up- 210710bb41e0 Zhamak Dehghani | Kafka Summit Europe 2021 Keynote: How to Build the Data Mesh Foundation https://www.youtube.com/watch?v=QF41q10NSAs https://datameshlearning.github.io/intro-to-data-mesh/

Data Mesh for Dinner

Data Mesh for Dinner

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados(20)

Similar a Data Mesh for Dinner

Similar a Data Mesh for Dinner(20)

Mais de Kent Graziano

Mais de Kent Graziano(20)

Último

Último(20)

Data Mesh for Dinner