Lessons Learned Building a Connector Using Kafka Connect (Katherine Stanley & Andrew Schofield, IBM UK) Kafka Summit SF 2019

Lessons learned building a connector
using Kafka Connect
Kate Stanley and Andrew Schofield
Kafka Summit SF 2019
IBM Event StreamsApache Kafka© 2019 IBM Corporation

“Kafka Connect is a tool for scalably and
reliably streaming data between Apache
Kafka and other systems”
© 2019 IBM Corporation

IBM MQ

MESSAGE QUEUING EVENT STREAMING
Assured delivery Stream history

IBM MQ
MQ
CLIENT
APP
MQ
CLIENT
APP
MQ
CLIENT
APP
MQ
CLIENT
APP

KAFKA
CLIENT
APP
KAFKA
CLIENT
APP
KAFKA CONNECT
IBM MQ
MQ
CLIENT
APP
MQ
CLIENT
APP
MQ
CLIENT
APP
MQ
CLIENT
APP

Getting started with Kafka
Connect

Kafka Connect – an API for
implementing connectors

Kafka Connect – a runtime for executing
connector processes

Connector

Connector
kafka-connect-mq-source.jar

Connector
Connect worker
CLASSPATH

Connector
Connect worker
CLASSPATH
Connector X
Task X
Task X

Connector
Connect worker
CLASSPATH
Connector X
Task X
Task X
Connector plugin
Connector

FROM ibmcom/eventstreams-kafka
COPY connect-distributed.properties /opt/kafka/config/
COPY connectors /opt/connectors/
WORKDIR /opt/kafka
EXPOSE 8083
ENTRYPOINT ["./bin/connect-distributed.sh",
"config/connect-distributed.properties"]

$ connect-distributed.sh
connect-distributed.properties
$ connect-standalone.sh
connect-standalone.properties
connector1.properties

$ curl http://localhost:8083/connector-plugins
[
{
"class":"org.apache.kafka.connect.file.FileStreamSinkConnector",
"type":"sink",
"version":"2.3.0”
},
{
"class":"org.apache.kafka.connect.file.FileStreamSourceConnector",
"type":"source",
"version":"2.3.0”
}
]

$ echo ‘{
"name":"kate-file-load",
"config":{"connector.class":"FileStreamSource",
"file":"config/server.properties",
"topic":"kafka-config-topic"}}’ |
curl -X POST -d @- http://localhost:8083/connectors
--header "content-Type:application/json"
$ curl http://localhost:8083/connectors
["kate-file-load"]

Connector internals

Anatomy of a connector
Task X
Connector X Task X
Task X

Anatomy of a connector

Key considerations – partitions, tasks and topics

file.txt
1. Start
2. The beginning
3. The middle
4. Conclusion
5. Ending
6. Finish

Partition 1
file.txt
1. Start
2. The beginning
3. The middle
4. Conclusion
5. Ending
6. Finish
1. Start
3. The middle
5. Ending
2. The beginning
4. Conclusion
6. Finish
Partition 2SOURCE
CONNECTOR
Topic

file-copy.txt
Partition 1
file.txt
1. Start
2. The beginning
3. The middle
4. Conclusion
5. Ending
6. Finish
1. Start
3. The middle
5. Ending
2. The beginning
4. Conclusion
6. Finish
Partition 2SOURCE
CONNECTOR
SINK
CONNECTOR
1. Start
3. The middle
5. Ending
2. The beginning
4. Conclusion
6. Finish
Topic

Key considerations – Data formats

EXTERNAL SYSTEM
FORMAT
KAFKA RECORD
FORMAT
KAFKA CONNECT INTERNAL
FORMAT

EXTERNAL SYSTEM
FORMAT
KAFKA RECORD
FORMAT
FORMAT
Connector builders
e.g. com.ibm.eventstreams.connect.mqsource.builders.JsonRecordBuilder
e.g. com.ibm.eventstreams.connect.mqsink.builders.JsonMessageBuilder

org.apache.kafka.connect.converters.ByteArrayConverter
org.apache.kafka.connect.storage.StringConverter
org.apache.kafka.connect.json.JsonConverter
EXTERNAL SYSTEM
FORMAT
KAFKA RECORD
FORMAT
FORMAT

Implementing the API

version()
config()
validate(config)
start(config)
Connector initialize
parse and
validate config
Lifecycle of a connector

Connector config
@Override
public ConfigDef config() {
ConfigDef configDef = new ConfigDef();
configDef.define(”config_option", Type.STRING, Importance.HIGH, ”Config option.");
return configDef;
}
$ curl -X PUT -d '{"connector.class":”MyConnector"}’
http://localhost:8083/connector-plugins/MyConnector/config/validate
{“configs”: [{
“definition”: {“name”: “config_option”, “importance”: “HIGH”, “default_value”: null, …},
”value”: {
“errors”: [“Missing required configuration ”config_option” which has no default value.”],
…
}

version()
config()
validate(config)
start(config)
taskClass()
taskConfigs(max)
initialize
parse and
validate config create tasks
stop()
Connector

Source Task initialize running
stop()
poll()
commit()
commitRecord(record)
version()
start(config)
initialize
parse and
validate config create tasks
Connector

initialize running
stop()
put(records)
flush(offsets)
version()
start(config)
Sink Task
initialize
parse and
validate config create tasksConnector

Kafka Connect and IBM MQ

It’s easy to connect IBM MQ to Apache Kafka
IBM has created a pair of connectors, available as source code or as part of IBM Event
Streams
Source connector
From MQ queue to Kafka topic
https://github.com/ibm-messaging/kafka-connect-mq-source
Sink connector
From Kafka topic to MQ queue
https://github.com/ibm-messaging/kafka-connect-mq-sink
Fully supported by IBM for customers with support entitlement for IBM Event Streams

Where can I get these magnificent connectors?
https://ibm.github.io/event-streams/connectors/

Connecting IBM MQ to Apache Kafka
The connectors are deployed into a Kafka Connect runtime
This runs between IBM MQ and Apache Kafka
CLIENT
IBM MQ
TO.KAFKA.Q
FROM.KAFKA.Q
Kafka Connect worker
FROM.MQ.TOPIC
MQ SINK
CONNECTOR
TO.MQ.TOPIC
MQ SOURCE
CONNECTOR
CLIENT

Running Kafka Connect on a mainframe
IBM MQ Advanced for z/OS VUE provides support for the Kafka Connect workers to
be deployed onto z/OS Unix System Services using bindings connections to MQ
BINDINGS
IBM MQ Advanced
for z/OS VUE
TO.KAFKA.Q
FROM.KAFKA.Q
FROM.MQ.TOPIC
MQ SINK
CONNECTOR
TO.MQ.TOPIC
MQ SOURCE
CONNECTOR
BINDINGS
Unix System Services

Design of the MQ sink
connector

MQ sink connector
Converter MessageBuilder
TO.MQ.TOPIC SinkRecord
Value
(may be complex)
Schema
Kafka Record
Value
byte[]
Key
MQ Message
Payload
MQMD
(MQRFH2)
MQ SINK
CONNECTOR
FROM.KAFKA.Q

Sink task – Design
Sink connector is relatively simple
The interface is synchronous and fits MQ quite well
Balancing efficiency with resource limits is the key
put(Collection<SinkRecord> records)
Converts Kafka records to MQ messages and sends in a transaction
Always requests a flush to avoid hitting MQ transaction limits
flush(Map<TopicPartition, OffsetAndMetadata> currentOffsets)
Commits any pending sends
This batches messages into MQ without excessively large batches

Design of the MQ source
connector

MQ source connector
RecordBuilder Converter
TO.MQ.TOPICSource Record
Value
(may be complex)
Schema
MQ Message Kafka Record
Null Record
MQ SOURCE
CONNECTOR
TO.KAFKA.Q
Value
byte[]
Payload
MQMD
(MQRFH2)

Source task – Original design
Source connector is more complicated
It’s multi-threaded and asynchronous which fits MQ less naturally
List<SourceRecord> poll()
Waits for up to 30 seconds for MQ messages and returned as a batch
Multiple calls to poll() could contribute to an MQ transaction
commit()
Asynchronously commits the active MQ transaction
Works quite well but commit() is too infrequent under load which causes throttling
commit() does ensure that the most recent batch of messages polled have been acked by Kafka, but it
doesn’t quite feel like the right way to do it

Source task – Revised design
Changed so each call to poll() comprises a single MQ transaction
commit() is no longer used in normal operation
List<SourceRecord> poll()
Waits for records from the previous poll() to be acked by Kafka
Commits the active MQ transaction – the previous batch
Waits for up to 30 seconds for MQ messages and returned as a new batch
commitRecord(SourceRecord record)
Just counts up the acks for the records sent
MQ transactions are much shorter-lived
No longer throttles under load
Feels a much better fit for the design of Kafka Connect

Stopping a source task is tricky
stop() is called on SourceTask to indicate the task should stop
Running asynchronously wrt to the polling and commit threads
Can’t be sure whether poll() or commit() are currently active or will be called very soon
Since poll() and commit() may both want access to the MQ connection
It’s not clear when it’s safe to close it
KIP-419: Safely notify Kafka Connect SourceTask is stopped
Adds a stopped() method to SourceTask that is guaranteed to be the final call to the task
uninitialized
initialize()
initialized running stopping
start() stop() stopped()
poll()
commit()
commitRecord()
poll()
commit()
commitRecord()© 2019 IBM Corporation

Summary
Over 80 connectors
IBM MQ
HDFS
Elasticsearch
MySQL
JDBC
MQTT
CoAP
+ many others

Summary
Connector initialize
parse and
validate config
create tasks
Sink Task initialize running
Source Task initialize running

Summary

Thank you
Kate Stanley
Andrew Schofield
Kafka Connect:
IBM Event Streams:
@katestanley91
https://medium.com/@andrew_schofield
https://kafka.apache.org/documentation/#connect
https://github.com/ibm-messaging/kafka-connect-mq-source
https://github.com/ibm-messaging/kafka-connect-mq-sink
https://ibm.github.io/event-streams/connectors/
ibm.com/cloud/event-streams

Lessons Learned Building a Connector Using Kafka Connect (Katherine Stanley & Andrew Schofield, IBM UK) Kafka Summit SF 2019

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Lessons Learned Building a Connector Using Kafka Connect (Katherine Stanley & Andrew Schofield, IBM UK) Kafka Summit SF 2019

Semelhante a Lessons Learned Building a Connector Using Kafka Connect (Katherine Stanley & Andrew Schofield, IBM UK) Kafka Summit SF 2019 (20)

Mais de confluent

Mais de confluent (20)

Último

Último (20)

Lessons Learned Building a Connector Using Kafka Connect (Katherine Stanley & Andrew Schofield, IBM UK) Kafka Summit SF 2019