As your organization rapidly grows in scale, so do the amount of challenges. Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc. Coming up with an architecture that can handle all of the data flows in a universal, simple way is key. This talk is about Wix's Kafka based global data architecture and platform. How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use. All the while offering various SDKs (some of them open-source), tools, and features for adapting to growing scale and insuring high resilience.

1. @NSilnitsky
@NSilnitsky
Kafka- based Global Data Mesh
Kafka-based Global Data
Mesh at Wix
Natan Silnitsky Backend Infra TL, Wix.com
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil

2. @NSilnitsky
Kafka- based Global Data Mesh

3. @NSilnitsky
@NSilnitsky
Kafka- based Global Data Mesh
Enable any business,
community or person on earth
to make their online dream
come true.
Our mission

4. @NSilnitsky
Kafka- based Global Data Mesh
At Wix
±1B
Unique
visitors
>100M
Websites
Published by
Wix users
(5-7% of all
internet
websites)
±6000
People
Work at Wix
use Wix
platform
every month

5. @NSilnitsky
500B >600
GAs Per Day
~1500
Developers
Daily HTTP
Transactions
18
Data
Centers &
Pops
2
Cloud Providers
[Google/AWS]
Scaling out
Kafka- based Global Data Mesh

6. @NSilnitsky
Wix has over 2300 microservices in production
Kafka- based Global Data Mesh

7. @NSilnitsky
@NSilnitsky
A Fully Distributed Environment
70B daily produced Events
4 Regions @ GCP/AWS
~50k Topics
> 500k Partitions
25% of the data is replicated
+ We assume
x10 scale factor
during 12-24 next
months

8. @NSilnitsky
@NSilnitsky
Growing
Pains
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Cluster
overload
DC
awareness
* keep simple even. Migrated, disributed

9. @NSilnitsky
@NSilnitsky
Growing
Pains
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Cluster
overload
DC
awareness
* messaging platform

10. @NSilnitsky
@NSilnitsky
Growing
Pains
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Cluster
overload
DC
awareness
* tackled blockers

11. @NSilnitsky
@NSilnitsky
Event-driven
architecture
Kafka
Growing
Pains
Large scale
traffic
Hundreds of
developers
Diverse
tech stack
Cluster
overload
DC
awareness

12. @NSilnitsky
Kafka Broker
You Know
Kafka
Kafka
Consumer
Kafka Broker
Service A Service B
Kafka
Producer
Kafka- based Global Data Mesh

13. @NSilnitsky
If You Don’t
Know Kafka
Kafka- based Global Data Mesh
Kafka Broker
Kafka Broker
Topic
Partition

14. @NSilnitsky
Kafka
Broker Topic
Kafka- based Global Data Mesh
Partition
Kafka
Producer
0 1 2 3 4 5
Service A

15. @NSilnitsky
Kafka
Broker Topic
Kafka- based Global Data Mesh
Partition
Kafka
Producer
0 1 2 3 4 5
Service A
Service B
Kafka
Consumer

16. @NSilnitsky
@NSilnitsky
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Kafka
Growing
Pains
Cluster
overload
DC
awareness

17. @NSilnitsky
Greyhound
Consumer
Greyhound
Producer
Greyhound*
Wraps
Kafka
* open source software
Kafka Broker
Kafka- based Global Data Mesh
Kafka
Producer
Service A Service B
Kafka
Consumer

18. @NSilnitsky
Greyhound
Consumer
Greyhound
Producer
Kafka Broker
Kafka- based Global Data Mesh
Kafka
Producer
Service A Service B
Kafka
Consumer

19. @NSilnitsky
@NSilnitsky
aGreyhoundConsumerSpec(groupName, messageHandler, topicName)
val producer = GreyhoundBuilder.resilientProducer(topicName)
Kafka- based Global Data Mesh

20. @NSilnitsky
✔ Parallel Consumption
✔ Batch Consumer
✔ Consumer Retry Strategies
✔ Resilient Producer
✔ Context Propagation
✔ Metrics reporting
Self Service 🟊
Greyhound
Kafka- based Global Data Mesh
Kafka Broker

21. @NSilnitsky
Scala NodeJS / Python / Go
Greyhound
Kafka Broker
I’m a container
gRPC + proto
sidecar
JVM library
Kafka- based Global Data Mesh

22. @NSilnitsky
Kafka Broker
Domain Events
ETL to DWH
Task Scheduling
Async Queue
...
We use it
for...
Kafka- based Global Data Mesh
Infra

23. @NSilnitsky
@NSilnitsky
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Kafka + Greyhound
Growing
Pains
Large scale
traffic
Cluster
Overload
DC
awareness

24. @NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
Kafka Broker
Single
Cluster
Overload
* intra-dc

25. @NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
Single
Cluster
Overload
A
A
A
A
B
Kafka Cluster (by SLAs)
B
Multi
Cluster
C

26. @NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
A
A
A
A
B
To which
Cluster?
Kafka Cluster (by SLAs)
B
I want to produce
a domain event.
C

27. @NSilnitsky
@NSilnitsky
aGreyhoundConsumerSpec(groupName, messageHandler, topicName)
.ClusterA
val producer = GreyhoundBuilder.resilientProducer(topicName,
_.ClusterB
Kafka- based Global Data Mesh
* wix-layer

28. @NSilnitsky
Cluster A + DC1 Cluster A + DC2

29. @NSilnitsky
@NSilnitsky
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Kafka + Greyhound
Growing
Pains
Large scale
traffic
Cluster
Overload
DC
awareness
+ Multi Cluster

30. @NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
Master Active
Master Passive
Read-Only DBs
Http/gRPC
To which
DC?
I want to
process the event
and save the data.

31. @NSilnitsky
@NSilnitsky
Kafka + Greyhound
+ Cross DC services
Kafka- based Global Data Mesh
Growing
Pains
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Large scale
traffic
Cluster
Overload
DC
awareness
+ Multi Cluster

32. @NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. Replicate across DCs
2. Produce to Main DC
[WIP] Just Produce ❤
Greyhound
Kafka

33. @NSilnitsky
Replicate !
DC1 DC2
across DCs
Greyhound
producer
Greyhound
consumer
Kafka
Broker
Cluster
A
Kafka
Broker
Cluster
Kafka- based Global Data Mesh
A

34. @NSilnitsky
Greyhound
Batch
Consumer
Greyhound
Producer
Batch #
Replicate !
Kafka
Broker
Cluster
A
Kafka
Broker
Cluster
DC1 DC2
How?
Greyhound
producer
Greyhound
consumer
Kafka- based Global Data Mesh
A
Cross DC
Replication
Service

35. @NSilnitsky
Replicate !
DC1 DC2
a Compact Topic for Policy
A
A
Kafka
Broker
Cluster
Kafka
Broker
Cluster
Greyhound
producer
Greyhound
consumer
Kafka- based Global Data Mesh

36. @NSilnitsky
Replication
service
Replicate !
DC1 DC2
How?
A
KVStoreReader
In-Memory Map
Key Value
Topic1 AllDCs
Topic2 Custom
A
Kafka
Broker
Cluster
Kafka
Broker
Cluster
Greyhound
producer
Greyhound
consumer
Kafka- based Global Data Mesh

37. @NSilnitsky
Kafka- based Global Data Mesh

38. @NSilnitsky
Kafka- based Global Data Mesh

39. @NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. ✔ Replicate across DCs
2. Produce to Main DC
[WIP] Just Produce ❤
Greyhound
Kafka

40. @NSilnitsky
Produce !
DC1 DC2
To main DC
Read-Only DB /
Master Passive DB
Master Active
A
Kafka
Broker
Cluster
Kafka- based Global Data Mesh
Greyhound
producer

41. @NSilnitsky
main-dc-producer
service
Produce !
DC1 DC2
How?
Master Active
Kafka- based Global Data Mesh
Greyhound
gRPC
producer
producer(topicName, _.toMainDc)

42. @NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. ✔ Replicate across DCs
2. ✔ Produce to Main DC
[WIP] Just Produce ❤
Greyhound
Kafka

43. @NSilnitsky
DC1 DC2
To where needed
[WIP] Just Produce ❤
Kafka- based Global Data Mesh
Greyhound
producer
Greyhound
consumer
Master Active

44. @NSilnitsky
Topology
Topic List
[consumer
group:DC]
DC1 DC2
A
Kafka
Broker
Cluster
Kafka- based Global Data Mesh
Greyhound
producer
Greyhound
consumer
To where needed
[WIP] Just Produce ❤
Master Active

45. @NSilnitsky
Topology
Topic List
[consumer
group:DC]
DC1 DC2
A
Kafka
Broker
Cluster
Master Active
Kafka- based Global Data Mesh
Greyhound
producer
Greyhound
consumer
To where needed
[WIP] Just Produce ❤
Replication service
* reintroduce decoupling

46. @NSilnitsky
@NSilnitsky
Kafka- based Global Data Mesh
DC1 DC2
DC3 DC4
A
A
A
A
B
B
Just Produce ❤
To a Single Logical
Kafka Cluster.
Kafka Broker
Cluster
DB

47. @NSilnitsky
@NSilnitsky
Kafka- based Global Data Mesh
Any
Tech Stack
Greyhound
Anywhere
You Need
developer

48. @NSilnitsky
developer
github.com/wix/greyhound
Kafka- based Global Data Mesh

49. @NSilnitsky
@NSilnitsky
Thank
You!
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
👉 slideshare.net/NatanSilnitsky
Any questions?
Kafka- based Global Data Mesh

50. @NSilnitsky
@NSilnitsky
Q&A
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
👉 slideshare.net/NatanSilnitsky
Any questions?
Kafka- based Global Data Mesh