Webcast available here: https://videos.confluent.io/watch/ptpxz9ks5cratppU9d4bUD? Speaker: Robin Moffatt

1. From Zero to Hero
with Kafka Connect
A practical guide to becoming l33t with Kafka Connect
a.k.a.

2. @rmoff | @confluentinc
• Robin Moffatt (@rmoff)
• Senior Developer Advocate at Confluent
• Working in data & analytics since 2001
• Oracle ACE Director (Alumnus)
$ whoami
http://rmoff.dev/talks · http://rmoff.dev/blog · http://rmoff.dev/youtube

3. @rmoff | @confluentinc
• Slides: yes!
• Recording: yes!
• Questions: yes!
Housekeeping

4. @rmoff | @confluentinc
What is
Kafka
Connect?

5. @rmoff | @confluentinc
Sources
Streaming Integration with Kafka Connect
Kafka Brokers
Kafka Connect
syslog

6. @rmoff | @confluentinc
Streaming Integration with Kafka Connect
Kafka Brokers
Kafka Connect
Amazon S3
Google BigQuery
Sinks

7. @rmoff | @confluentinc
Streaming Integration with Kafka Connect
Kafka Brokers
Kafka Connect
syslog
Amazon S3
Google BigQuery

8. @rmoff | @confluentinc
{
"connector.class":
"io.confluent.connect.jdbc.JdbcSourceConnector",
"connection.url":
"jdbc:mysql://asgard:3306/demo",
"table.whitelist":
"sales,orders,customers"
}
https://docs.confluent.io/current/connect/
Look Ma, No Code!

9. @rmoff | @confluentinc
Streaming Pipelines
RDBMS Kafka
Connect
Kafka
Connect
Amazon S3
Splunk
e.g. Postgres,
Oracle

10. @rmoff | @confluentinc
KafkaConnect
Writing to data stores from Kafka
App
Data
Store

11. @rmoff | @confluentinc
Evolve processing from old systems to new
RDBMS
Existing
App
New App
<x>
Kafka
Connect
e.g. Postgres,
Oracle

12. @rmoff | @confluentinc
Demo
http:!//rmoff.dev/kafka-connect-code

13. @rmoff | @confluentinc
Configuring
Kafka
Connect
Inside the API - connectors, transforms, converters

14. @rmoff | @confluentinc
Kafka Connect basics
KafkaKafka ConnectSource

15. @rmoff | @confluentinc
Connectors
KafkaKafka ConnectSource
Connector

16. @rmoff | @confluentinc
Connectors
"config": {
[...]
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"connection.url": "jdbc:postgresql://postgres:5432/",
"topics": "asgard.demo.orders",
}

17. @rmoff | @confluentinc
Connectors
Connect
Record
Native data
Connector
KafkaKafka ConnectSource

18. @rmoff | @confluentinc
Converters
Connect
Record
Native data bytes[]
KafkaKafka ConnectSource
Connector Converter

19. @rmoff | @confluentinc
Serialisation & Schemas
https://qconnewyork.com/system/files/presentation-slides/qcon_17_-_schemas_and_apis.pdf
Avro Protobuf JSON CSVJSON
Schema
👍 👍 👍 😬

20. @rmoff | @confluentinc
The Confluent Schema Registry
Source
Avro
Message
Target
Schema
RegistryAvro
Schema
Kafka
Connect
Kafka
ConnectAvro
Message

21. @rmoff | @confluentinc
Converters
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://localhost:8081
value.converter=org.apache.kafka.connect.storage.StringConverter
Set as a global default per-worker; optionally can be overriden per-connector

22. @rmoff | @confluentinc
Single Message Transforms
KafkaKafka ConnectSource
Connector
Transform(s)
Converter

23. @rmoff | @confluentinc
Single Message Transforms
"config": {
[...]
"transforms": "addDateToTopic,labelFooBar",
"transforms.addDateToTopic.type": "org.apache.kafka.connect.transforms.TimestampRouter",
"transforms.addDateToTopic.topic.format": "${topic}-${timestamp}",
"transforms.addDateToTopic.timestamp.format": "YYYYMM",
"transforms.labelFooBar.type": "org.apache.kafka.connect.transforms.ReplaceField$Value",
"transforms.labelFooBar.renames": "delivery_address:shipping_address",
}
Do these transforms
Transforms config Config per transform

24. @rmoff | @confluentinc
Extensible
Connector
Transform(s)
Converter

25. @rmoff | @confluentinc
Confluent Hub
hub.confluent.io

26. @rmoff | @confluentinc

27. @rmoff | @confluentinc

28. @rmoff | @confluentinc

29. @rmoff | @confluentinc

30. @rmoff | @confluentinc

31. @rmoff | @confluentinc

32. Integrating
Databases
and Apache
Kafka®
https://rmoff.dev/xbf

33. @rmoff | @confluentinc
Deploying
Kafka
Connect
Connectors, Tasks, and Workers

34. @rmoff | @confluentinc
Connectors and Tasks
JDBC Source S3 Sink
JDBC Task #2JDBC Task #1
S3 Task #1

35. @rmoff | @confluentinc
Connectors and Tasks
JDBC Source S3 Sink
JDBC Task #2JDBC Task #1
S3 Task #1

36. @rmoff | @confluentinc
Connectors and Tasks
JDBC Source S3 Sink
JDBC Task #2JDBC Task #1
S3 Task #1

37. @rmoff | @confluentinc
Tasks and Workers
JDBC Source S3 Sink
JDBC Task #2JDBC Task #1
S3 Task #1
Worker

38. @rmoff | @confluentinc

39. @rmoff | @confluentinc
Kafka Connect Standalone Worker
JDBC Task #2JDBC Task #1
S3 Task #1
Worker
Offsets

40. @rmoff | @confluentinc
"Scaling" the Standalone Worker
JDBC Task #2
JDBC Task #1
S3 Task #1
Worker
OffsetsOffsets
Worker
Fault-tolerant? Nope.

41. @rmoff | @confluentinc
JDBC Task #2
Kafka Connect Distributed Worker
JDBC Task #1 JDBC Task #2
S3 Task #1
Offsets
Config
Status
Fault-tolerant? Yeah!
Worker
Kafka Connect cluster

42. @rmoff | @confluentinc
Scaling the Distributed Worker
JDBC Task #1 JDBC Task #2
S3 Task #1
Offsets
Config
Status
Fault-tolerant? Yeah!
Worker Worker
Kafka Connect cluster

43. @rmoff | @confluentinc
Distributed Worker - fault tolerance
JDBC Task #1
S3 Task #1
Offsets
Config
Status
Worker Worker
Kafka Connect cluster

44. @rmoff | @confluentinc
Distributed Worker - fault tolerance
JDBC Task #1
S3 Task #1
Offsets
Config
Status
Worker
Kafka Connect cluster
JDBC Task #2

45. @rmoff | @confluentinc
Multiple Distributed Clusters
JDBC Task #1
S3 Task #1
Offsets
Config
Status
Kafka Connect cluster #1
JDBC Task #2
Kafka Connect cluster #2
Offsets
Config
Status

46. @rmoff | @confluentinc
Containers

47. @rmoff | @confluentinc
Kafka Connect images on Docker Hub
confluentinc/cp-kafka-connect-base
kafka-connect-elasticsearch
kafka-connect-jdbc
kafka-connect-hdfs
[…]
confluentinc/cp-kafka-connect

48. @rmoff | @confluentinc
Adding connectors to a container
confluentinc/cp-kafka-connect-base
JAR
Confluent Hub

49. @rmoff | @confluentinc
At runtime
JAR
confluentinc/cp-kafka-connect-base
kafka-connect:
image: confluentinc/cp-kafka-connect:5.5.0
environment:
CONNECT_PLUGIN_PATH: '/usr/share/java,/usr/share/confluent-hub-components'
command:
- bash
- -c
- |
confluent-hub install --no-prompt neo4j/kafka-connect-neo4j:1.0.0
/etc/confluent/docker/run
http://rmoff.dev/ksln19-connect-docker

50. @rmoff | @confluentinc
Build a new image
FROM confluentinc/cp-kafka-connect:5.5.0
ENV CONNECT_PLUGIN_PATH="/usr/share/java,/usr/share/confluent-hub-components"
RUN confluent-hub install --no-prompt neo4j/kafka-connect-neo4j:1.0.0
JAR
confluentinc/cp-kafka-connect-base

51. @rmoff | @confluentinc
# Launch Kafka Connect
/etc/confluent/docker/run &
#
# Wait for Kafka Connect listener
while [ $$(curl -s -o /dev/null -w %{http_code} http:"//$$CONNECT
echo -e $$(date) " Kafka Connect listener HTTP state: " $$(cur
sleep 5
done
#
# Create JDBC Source connector
curl -X POST http:"//localhost:8083/connectors -H "Content-Type:
"name": "jdbc_source_mysql_00",
"config": {
"connector.class": "io.confluent.connect.jdbc.
"connection.url": "jdbc:mysql:"//mysql:3306/dem
http://rmoff.dev/ksln19-connect-docker
Automating connector creation

52. @rmoff | @confluentinc
Troubleshooting
Kafka Connect

53. @rmoff | @confluentinc
Troubleshooting Kafka Connect
$ curl -s "http://localhost:8083/connectors/source-debezium-orders/status" |
jq '.connector.state'
"RUNNING"
$ curl -s "http://localhost:8083/connectors/source-debezium-orders/status" |
jq '.tasks[0].state'
"FAILED"
http://go.rmoff.net/connector-status
RUNNING
FAILED
Connector
Task

54. @rmoff | @confluentinc
Troubleshooting Kafka Connect
curl -s "http:!//localhost:8083/connectors/source-debezium-orders-00/status"
| jq '.tasks[0].trace'
"org.apache.kafka.connect.errors.ConnectExceptionntat
io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)ntat
io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:197)ntat
io.debezium.connector.mysql.BinlogReader$ReaderThreadLifecycleListener.onCommunicationFailure(BinlogReader.java:1018)nt
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:950)ntat
com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580)ntat
com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:825)ntat
java.lang.Thread.run(Thread.java:748)nCaused by: java.io.EOFExceptionntat
com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:190)ntat
com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readInteger(ByteArrayInputStream.java:46)ntat
com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserializer.java
:35)ntat
com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserializer.java
:27)ntat
com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:212)ntat
io.debezium.connector.mysql.BinlogReader$1.nextEvent(BinlogReader.java:224)ntat
com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:922)nt!!... 3 moren"

55. @rmoff | @confluentinc
The log is the source of truth
$ confluent local log connect
$ docker-compose logs kafka-connect
$ cat /var/log/kafka/connect.log

56. @rmoff | @confluentinc
Dynamic log levels
https://rmoff.dev/kc-dynamic-log-level
curl -s http://localhost:8083/admin/loggers/ | jq
{
"org.apache.kafka.connect.runtime.rest": {
"level": "WARN"
},
"org.reflections": {
"level": "ERROR"
},
"root": {
"level": "INFO"
}
}
curl -s -X PUT http://localhost:8083/admin/loggers/io.debezium
-H "Content-Type:application/json" -d '{"level": "TRACE"}'
(Added in Apache Kafka 2.4 / Confluent Platform 5.4)

57. @rmoff | @confluentinc
Error Handling
and
Dead Letter
Queues

58. @rmoff | @confluentinc
org.apache.kafka.common.errors.SerializationException:
Unknown magic byte!

59. @rmoff | @confluentinc
Mismatched converters
"value.converter":
"AvroConverter"
org.apache.kafka.common.errors.SerializationException:
Unknown magic byte!
Use the correct Converter for the
source dataⓘ
Messages are not Avro

60. @rmoff | @confluentinc
Mixed serialisation methods
"value.converter":
"AvroConverter"
org.apache.kafka.common.errors.SerializationException:
Unknown magic byte!
Some messages are not Avro
Use error handling to deal
with bad messagesⓘ

61. @rmoff | @confluentinc
Error Handling and DLQ
Handled
Convert
-> read/write from Kafka
-> [de]-serialisation
Transform
Not Handled
Start
-> Connections to a data store
Poll / Put
-> Read/Write from/to data store*
* can be retried by Connect
https://cnfl.io/connect-dlq

62. @rmoff | @confluentinc
Fail Fast
Kafka Connect
Source topic messages
Sink messages
https://cnfl.io/connect-dlq

63. @rmoff | @confluentinc
YOLO ¯_(ツ)_/¯
Kafka Connect
Source topic messages
Sink messages
errors.tolerance=all
https://cnfl.io/connect-dlq

64. @rmoff | @confluentinc
Dead Letter Queue
Kafka Connect
Source topic messages
Sink messages
Dead
letter
queue
errors.tolerance=all
errors.deadletterqueue.topic.name=my_dlq
https://cnfl.io/connect-dlq

65. @rmoff | @confluentinc
Re-processing the Dead Letter Queue
Source topic messages
Sink messages
Dead
letter
queue
Kafka Connect (Avro sink)
Kafka Connect (JSON sink)
https://cnfl.io/connect-dlq

66. @rmoff | @confluentinc
Metrics
and
Monitoring

67. @rmoff | @confluentinc
REST API
http://go.rmoff.net/connector-status

68. @rmoff | @confluentinc
Confluent Control Center

69. @rmoff | @confluentinc
Consumer lag

70. @rmoff | @confluentinc
JMX

71. @rmoff | @confluentinc
Standby for
resource
links…

72. @rmoff | @confluentinc
Free Books!
https://rmoff.dev/kp

73. @rmoff | @confluentinc
Free Training!
https://rmoff.dev/kp_train

74. @rmoff | @confluentinc
Kaiser Resource Contacts
Shaun Wanford, Integration Architect (Kafka as a Service SME)
shaun.x.wanford@kp.org
503-309-8849
Scott Bramhall, Kaiser Account Executive
scott.bramhall@confluent.io
925-899-8939 – mobile
Neil Hudson, Kaiser Sales Engineer
neil.Hudson@confluent.io
303-257-1064

75. #EOFhttps://talks.rmoff.net
@rmoff