Zalando transitioned from a centralized data platform to a data mesh architecture. This decentralized their data infrastructure by having individual domains own datasets and pipelines rather than a central team. It provided self-service data infrastructure tools and governance to enable domains to operate independently while maintaining global interoperability. This improved data quality by making domains responsible for their data and empowering them through the data mesh approach.

1. Data Mesh in Practice
Max Schultze - max.schultze@zalando.de
Arif Wider - awider@thoughtworks.com
17-11-2020
How Europe’s Leading
Online Platform for Fashion
Goes Beyond the Data Lake
@mcs1408 @arifwider

2. 2
Max Schultze
● Lead Data Engineer
● MSc in Computer Science
● Took part in early
development of Apache Flink
● Retired semi-professional
Magic: the Gathering player
Who are we?
Arif Wider
● Software engineering professor (full)
at HTW Berlin, Germany
● Fellow technology consultant with
ThoughtWorks Germany (part-time)
● Former Head of AI at ThoughtWorks
● Coffee geek

3. 3
TABLE OF
CONTENTS
Zalando’s Data Platform
What’s this Data Mesh?
Data Mesh in Practice

4. 4
Zalando’s Data Platform

5. 5
Zalando’s Data Platform
Ingestion
Storage
Serving

6. 6
Web
Tracking
Event Bus
DWH
Ingestion
Storage
Serving
Zalando’s Data Platform

7. 7
Web
Tracking
Event Bus
DWH
Ingestion
Storage
Serving
Metastore
Zalando’s Data Platform

8. 8
Web
Tracking
Event Bus
Ingestion
Storage
Serving
Metastore
Processing Platform
Fast Query Layer
DWH
Data
Catalog
Zalando’s Data Platform

9. 9
Centralization Challenges
Datasets provided by central data infrastructure team
● Lack of ownership
?

10. 10
Field_A Field_B
Record_1
Record_2
Record_3
Datasets provided by central data infrastructure team
● Lack of ownership
Data pipelines operated by central data infrastructure team
● Lack of quality
Centralization Challenges

11. 11
Centralization Challenges
Datasets provided by central data infrastructure team
● Lack of ownership
Data pipelines operated by central data infrastructure team
● Lack of quality
Organizational scaling
● Central team becomes the bottleneck

12. 12
A Recurring Pattern

13. 13
A Recurring Pattern

14. 14
A Recurring Pattern

15. 15
A Recurring Pattern

16. 16
Why is that?
central
data platform

17. 17
Why is that?
checkout
service
checkout
events

18. 18
What is Data Mesh?
Old wine applied to new bottles…
→ Product Thinking
→ Domain-Driven Distributed Architecture
→ Infrastructure as a Platform
… creates value from Data
https://martinfowler.com/articles/data-monolith-to-mesh.html by Zhamak Dehghani

19. 19
Data as a Product
Data
Product
What is my market?
What are the desires of
my customers?
What “price” is justified?
How to do marketing?
What’s the USP?
Are my customers happy?

20. 20
Domain-Driven Distributed Architecture… applied to Data
Domain

21. 21
Domain-Driven Distributed Architecture… applied to Data
Domain
→
Aggregated
Domain

22. 22
Domain-Driven Distributed Architecture… applied to Data
Discoverable
Addressable
Self-describing
Trustworthy
Interoperable
Secure
Domain
→
Aggregated
Domain

23. 23
...backed by domain-agnostic self-service data infrastructure
Data Infra as a Platform
Discoverable
Addressable
Self-describing
Trustworthy
Interoperable
Secure
Domain
→
Aggregated
Domain

24. 24
It’s a mindset shift
FROM TO
Centralized ownership Decentralized ownership
Pipelines as first class concern Domain Data as first class concern
Data as a by-product Data as a Product
Siloed Data Engineering Team Cross-functional Domain-Data Teams
Centralized Data Lake / Warehouse Ecosystem of Data Products

25. 25
Data Mesh in Practice

26. 26
Recap:
● From Bottleneck to Infra Platform
Data Mesh in Practice
Data Infra as a Platform

27. 27
Recap:
● From Bottleneck to Infra Platform
● From Data Monolith to Interoperable Services
Data Mesh in Practice
Data Infra as a Platform
central
data
platform

28. 28
Data Lake Storage
Governance Layer
Central Services with Global Interoperability

29. 29
Data Lake Storage
Bring Your Own Bucket (BYOB)
Governance Layer

30. 30
Processing Platform
Simplify Data Processing
Data Lake Storage
Governance Layer

31. 31
Processing Platform
Simplify Data Sharing
Data Lake Storage
Governance Layer

32. 32
Central Services with Global Interoperability
Decentralized ownership does not imply decentralized infrastructure!
Interoperability is created through convenient solutions of a self service platform.
Decentral Storage Central Infrastructure
Decentral Ownership Central Governance

33. 33
Recap:
● Datasets provided through pipelines of central data infrastructure teams
Data Mesh in Practice
?

34. 34
How to Ensure Data Quality?
Make conscious decisions
● Opt-in instead of default storage

35. 35
How to Ensure Data Quality?
Make conscious decisions
● Opt-in instead of default storage
● Behavioral changes - data is a product

36. 36
Care About Your User!
● Classification of Usage

37. 37
Care About Your User!
● Classification of Usage
● Dedicate resources to
○ Understand usage
○ Ensure quality

38. 38
Some Numbers

39. 39
Some Numbers
● 40 teams using BYOB

40. 40
Some Numbers
● 40 teams using BYOB
● 100 teams using the processing platform
Processing Platform

41. 41
Some Numbers
● 40 teams using BYOB
● 100 teams using the processing platform
● First curated data teams
Data Products
On Data Products
On Data Products
Processing Platform

42. 42
Some Numbers
● 40 teams using BYOB
● 100 teams using the processing platform
● First curated data teams
● 0 operational effort for the central team
Data Products
On Data Products
On Data Products
Processing Platform

43. 43
Some Numbers
● 40 teams using BYOB
● 100 teams using the processing platform
● First curated data teams
● 0 operational effort for the central team
Data Products
On Data Products
On Data Products
Processing Platform
It’s a journey ;)

44. 44
It’s a Journey

45. 45
“Off the shelf” data tooling

46. 46
“Off the shelf” data tooling
De-centralized archiving

47. 47
“Off the shelf” data tooling
De-centralized archiving
De-centralized GDPR deletion tooling

48. 48
“Off the shelf” data tooling
Template driven data preparation
De-centralized archiving
De-centralized GDPR deletion tooling

49. 49
Data Mesh in Practice
How Europe’s Leading
Online Platform for Fashion
Goes Beyond the Data Lake
Max Schultze
max.schultze@zalando.de
@mcs1408
Arif Wider
awider@thoughtworks.com
@arifwider