REX How Jobteaser got rid of an old data dump job using change data capture (CDC) with Debezium and Kafka.

1. 1
Streaming database
events over Kafka
using Debezium

2. 2
https://medium.com/jobteaser-dev-team
About me and where I work
@knil_sama
JobTeaser
Preparing the new generation to reach its
full potential, embrace the future with
optimism and make its mark in the world
Clément Demonchy
Data engineering with:
Python, AWS, Kubernetes, Kafka,
and anything that works
We are hiring !

3. 3
The need
Applicative data is used to
● Recommend offers to student
● Measure KPIs for the company
● Enrich users’ experience

4. 4
Separated space between data and website
JobTeaser infra

5. 5
Content

6. 6
● Fetch all the data
● Follow schema updates
● Real time
● No impact on production performance
The requirements

7. 7
Log-based Change-Data-Capture (CDC)

8. 8
Debezium
Created by RedHat
Open source
Free
Supports SQL DBs
Compatible Kafka connect

9. 9
We just happen to have a Kafka
NOT THIS ONE

10. 10
Kafka connect

11. 11
Install a connector
Upload connector runnable
on kafka connect worker
Write connector
configuration
Push connector

12. 12
A few more configuration to go

13. 13
1. Create a user with the following rights
SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT
2. Update configuration
Changes on MySQL

14. 14
To avoid connection dropping/hanging
expire_logs_days setting is not working with RDS
RDS specific changes

15. 15
Increase workers memory
=> For initial snapshot surge (Database)
Connectors in distributed mode
=> Fully stateless
auto.create.topics.enable topic set to true
=> Large number of tables and new tables
Changes on Kafka

16. 16
snapshot.mode :
With data: initial, when_needed or never
Without data: schema_only, schema_only_recovery
On RDS you need a global lock during snapshot
otherwise use snapshot.locking.mode : minimal, extended none
Debezium Snapshot

17. 17
Good to go ?

18. 18
Tables Columns Content
Anonymization strategies with Debezium
table.whitelist/table.blacklist column.mask.with.length.chars
Always prefer explicit whitelisting
because you can’t prevent columns changing name or being added
column.blacklist column.whitelist

19. 19
In our case the pre-existing database was
~ 40 GB db
> 100 Tables
Initial snapshot took 40 minutes to complete
In practice

20. 20
Content
Pushing it in production

21. 21
Big row issue, we have row with a size greater than default value !!
In theory you only need of increasing max.request.size, but it’s not enough
On Kafka connects worker
On Kafka brokers
Issues with Debezium (part 1)

22. 22
Some DCL command
Make the connector crash
So we had to set in connector
Issues with Debezium (part 2)

23. 23
If MySQL goes down, connector will fail
and you have to restart the task
1) Identify the failing task
2) Restart it
Easy case: binlog and consumer offset still exist
Connector stream recovery

24. 24
Basic monitoring:
● Prometheus
● Grafana
● Alerting on Slack
Be less strict and log errors instead of crashing for
Watch out for DEBUG with some loggers else you will flood the worker
Staying alive

25. 25
Kafka output

26. 26
Confluent created connector
Stream back debezium event to PostgreSQL
Handle create, update and schema changes
But … deleted records are not removed in target database
(A PR was merged recently and new default is to delete them)
JDBC Connect to the rescue

27. 27
Debezium provides an option on JDBC connector
That will add a flag column “__deleted” on every tables
Other usefuls SMTs : RemoveNulls, MultiTimestampConverter
Single Message Transformations (SMTs)

28. 28
Final result

29. 29
Thanks for your attention
Gimme questions !
We are hiring !