1. Designing a Data Mesh with Kafka
Paul Makkar
Rahul Gulati

2. Capital markets products,
services and liquidity
Saxo Bank facilitation Distribution to clients
We distribute our capital markets
products and services to our clients
through our platforms and APIs
We unbundle the value chain through
our open architecture
We source the best ideas, products, liquidity
and services from the best providers
SaxoTraderPRO and
SaxoTraderGO for
self-directed traders
SaxoInvestor for
self-directed and
delegating investors
Outsourced capital
markets
infrastructure
Traders
Investors
Wholesale
Services
Execution and trading
Risk management
Market data
Custody and back-
office
Reporting
Integration
Client management
Cloud hosting and IT
Business management
FIX API
Open API
Margin & risk management
Market data connectivity
Custody
Client account structures
EOD files, FSSO & TENS
Trading platforms
CRM & CMS API
OMS / EMS
Broker connectivity
Clearing and
settlement
Regulatory reporting
Hosting services
We run one tech stack and one, global set of business processes
Scalable core engine
A global facilitator of capital markets products and services
Saxo’s business model “BaaS”
2

3. Page 3 // Confidential //
“Data mesh objective is to create a foundation for getting value from analytical data
and historical facts at scale”
Zhamak Dehghani (formally Thoughtworks)
Data Mesh

4. Page 4 // Confidential //
Analytics Data Plane
Operational Data Plane
Realm of Data Engineer
Realm of Domain Experts Value Creators from data

5. Page 5 // Confidential //
Domain-oriented data ownership
Self-service data tech as a platform
Data-as-a-product
Federated data governance
Data Mesh: https://martinfowler.com/articles/data-mesh-principles.html
Slack: data-mesh-learning.slack.com
Data Mesh Founding Principles

6. Page 6 // Confidential //
Operational Data Plane Analytics Data Plane
New Operational Plane: Event Mesh Kafka
Shift Left
Directly Mapped Historical Facts

7. Page 7 // Confidential //
Self-Service Kafka Topics and Connectors
Meshable Data Models (‘Data-as-a-product’)
Gitops Governance
Shift Left

8. Page 8 // Confidential //
Remaining Challenges
How do you get 500 topics to your cloud Data Lake?
Schema Apples and Oranges
Data Access and Aggregation
Event Mesh > Data Mesh

9. Page 9 // Confidential //
How do you get 500 topics to your cloud Data Lake?

10. Page 10 // Confidential //
High Level Requirements
All topics automatically mirrored to the cloud data lake
Exclude list aware
Offset Preserving
ACLs, Config synching
Tombstone Preserving

11. Page 11 // Confidential //
Current
On Premise
Target

12. Page 12 // Confidential //
Automatically Copy Topics to the Cloud Data Lake

13. Page 13 // Confidential //
Confluent Replicator
Source Destination
Consumer
Replicator
Producer

14. Page 14 // Confidential //
Replicators: Configuration
Domain A Domain B
// Azure Pipelines using gitops way

15. Page 15 // Confidential //
// Offsets not preserved across clusters
// Many connectors to manage
// No ACL synching between clusters
// Topic deletion hard
Other Challenges with Replicators

16. Page 16 // Confidential //
// Inter Broker Mirroring
// Offset Preserving
// ACL Synching
// Byte to Byte Copy
Enter: Cluster Linking

17. Page 17 // Confidential //
Cluster linking at Saxo
(Cluster Linking)
On Premise
B B

18. Page 18 // Confidential //
Exclude List

19. Page 19 // Confidential //
Exclude List
Domain C
Domain
A
Domain B
Policy Manager

20. Page 20 // Confidential //
Offset Preserving

21. Page 21 // Confidential //
Offset Preserving
Topics with globally consistent offsets
On-Prem Cloud
Cluster Linking

22. Page 22 // Confidential //
ACL Management

23. Page 23 // Confidential //
ACL & Configs Sync
Source Destination
User: svcdev
Topic:
engagement1
Topic:
engagement2
User: oauthid User: svcdev
Topic:
engagement1
Topic:
engagement2
User: oauthid
ACL Sync

24. Page 24 // Confidential //
Schema Apples and Oranges

25. Page 25 // Confidential //
Analytical Plane
Operational Plane
Protobuf/Avro Deep Dive: https://www.slideshare.net/gschmutz/big-data-data-lake-datenserialisierungsformate

26. Page 26 // Confidential //
Domain Mapping
Engagement
Trading
.
.
.
Counterparty
Engagement
Trading
.
.
.
Counterparty
One to One Mapping
Domain based topic storage in data lake file system

27. Page 27 // Confidential //
Kafka -> Azure Data Lake
Data Lake
Storage
/raw/Instruments/TopicA
/raw/Instruments/TopicB
…
/raw/Trading/TopicC
/raw/Trading/TopicD
Topic A
Topic B
Instruments
Topic C
Topic D
Trading
2x ADLS Connector
2x ADLS Connector

28. Page 28 // Confidential //
Preserving Tombstones

29. Page 29 // Confidential //
Tombstone Propagation with DB Connector
100
99
Compact Topic
99
100
Tombstone
Delete K
Delete K

30. Page 30 // Confidential //
Tombstone Propagation with Data Lake Connector
100
Compact Topic
99
100
Tombstone
Ignore
Delete K

31. Page 31 // Confidential //
Enter: Data Lake SMT
{key:1, value: {Id: 121}}
{key:2, value: null}
ADLS
Connector
{key:2, value:
“Tombstone”}
SMT
{key:2, value: {Id: 122}}

32. Page 32 // Confidential //
Data Lake SMT Design
SMT
Null-Value Message
• Replace null with “tombstone” string
• Enrich
1
2
Every Message
Enrich Message value
• Key
• Partition
• Offset
• Timestamp

33. Page 33 // Confidential //
Recap
Cluster link Connector
Deployment
On Prem Cloud Data Lake
Topic
Template
Event Mesh Data Mesh

34. Page 34 // Confidential //
Access Control and Aggregations
Historical Facts
Aggregations?
Access
Layer

35. Page 35 // Confidential //
Thank You