Confluent Cloud makes Devops engineers lives a lot more easier. Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge. In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge. These lessons include: 1. Automation, Automation, Automation - all the process has to be completely automated at such scale 2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad 3. First migrate test topics with relayed real traffic - So data will be real but will not effect production. 4. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions 5. Adapt to Confluent Cloud APIs - e.g. lag monitoring

1. 5 lessons learned for
Successful Migration to
Confluent Cloud
Natan Silnitsky
Backend Infra Developer, Wix.com
natans@wix.com twitter@NSilnitsky linkedin/natansilnitsky natansil.com

2. >200M
Registered users
(website builders)
from 190 countries
@NSilnitsky
At Wix
>2,000M
Kafka messages
produced / Day
2,000
Microservices

3. * easily manage

4. Data Center 2
Data Center 1
@NSilnitsky
Confluent Cloud
Kafka Cluster
On premise
Kafka
Cluster
I
Kafka
Cluster
II
Kafka
Cluster
III
Kafka
Cluster
IV
* AWS/GCP
Microservice
Microservice
2000

5. Automation, Automation,
Automation
01

6. A Scala/Java high-level SDK for Apache Kafka.
Greyhound
@NSilnitsky

7. Site Apps
Service
Kafka
Consumer
Kafka
Producer
Greyhound
wraps Kafka
Ecom Catalog
Service
Kafka Broker
Greyhound Producer Greyhound Consumer
@NSilnitsky

8. Data Center 2
Data Center 1
Microservice
@NSilnitsky
Confluent Cloud
Kafka Cluster
On premise
Kafka
Cluster
I
Kafka
Cluster
II
Kafka
Cluster
III
Kafka
Cluster
IV
* all at once?

9. Gradual over All at once
02

10. Data Center 2
Data Center 1
Kafka Cluster
On premise
Confluent Cloud
Microservice
HTTP
HTTP
Kafka
Cluster
I
Kafka
Cluster
II
Kafka
Cluster
III
Kafka
Cluster
IV
Option B
Replicator
Topic 5
Topic 4
Topic 3
Topic 2
Topic 1
Topic 6
Migration
Orchestrator

11. Data Center 2
Data Center 1
@NSilnitsky
Kafka Cluster
On premise
Confluent Cloud
Microservice
HTTP
HTTP
Kafka
Cluster
I
Kafka
Cluster
II
Kafka
Cluster
III
Kafka
Cluster
IV
Option B
Replicator
Topic 5
Topic 4
Topic 3
Topic 2
Topic 1
Topic 6
Migration
Orchestrator
* self service - no downtime

12. First migrate test topics
with relayed real traffic
03

13. Data Center 2
@NSilnitsky
Kafka Cluster
On premise
Confluent Cloud
Test app
Real Topic->Test Topic
Test Topics
Kafka
Cluster
I
Kafka
Cluster
II
Kafka
Cluster
III
Kafka
Cluster
IV
Migration
Orchestrator
Replicator

14. Only needed
topics/partitions
04

15. Data Center 2
Data Center 1
Microservice
@NSilnitsky
0 Bytes 100 MB 170 MB
Confluent Cloud

16. Adapt to Confluent
Cloud APIs
05

17. Data Center 2
Data Center 1
Offset lag
Monitoring
@NSilnitsky
Confluent Cloud
Kafka Cluster
On premise
__consumer_offsets __consumer_offsets
Kafka Admin SDK Kafka Admin SDK
Unavailable
Available

18. So, We used Greyhound
and dedicated orchestration
services to have automatic,
safe, and gradual migration
to Confluent Cloud
@NSilnitsky

19. A Scala/Java high-level SDK for Apache Kafka.
0.1.5 is out!
github.com/wix/greyhound

20. Thank You
natans@wix.com twitter @NSilnitsky linkedin/natansilnitsky github.com/natansil

21. Slides & More
slideshare.net/NatanSilnitsky
medium.com/@natansil
twitter.com/NSilnitsky
natansil.com