1. Learnings From Shipping
1000+ Streaming Data Pipelines
To Production
Hakan Lofcali, Stefan Sprenger
{hakan,stefan}@datacater.io

2. ‣We develop tools for developers working with streaming data
‣With Kafka, Kubernetes, and less than 5 developers, we built a platform that helped teams to
deploy more than 1,000 streaming data pipelines to production
‣Let’s take you on our journey and the tools we adopted, hurdles encountered, and solutions
found
‣Infra Space
‣Customer Solutions Space
2
WHAT WE DO / WHO WE ARE

3. 3
STREAMING DATA PIPELINES
Continuous Applications of Data Transformations
Kafka Topic
Kafka
Connect
Source
Connector
Kafka
Connect
Sink
Connector
Kafka Topic Kafka Streams

4. 4
STREAMING PIPELINES IN THE WILD
Customer communications in real-time
Actionable
data
Clickstream
data
Outbox
service
Raw data
Process
clickstream events

5. 5
GOALS FOR THIS TALK
Avoid common pitfalls in streaming ETL
‣How to operate streaming data pipelines in an efficient and robust manner?
‣How to deal with resource-leaking Kafka Connect connectors?
‣How to monitor and debug running pipelines?
‣What are ways to deal with large data sources or slow data sinks?
‣What is missing in today’s ecosystem for streaming to become a commodity?

6. How to operate streaming data pipelines in
an efficient and robust manner?
6

7. 7
SEPARATE BY NODE POOL
Kafka NodePool
Apache Kafka
Broker*
Strimzi Kafka
Operator
Control Plane Nodepool
Pod
We operate one K8s Cluster - Multiple Node pools
K8s StatefulSet
K8s Deployment
DataCater
Control Plane
K8s Deployment
…
Kafka Connect Nodepool
Quarkus
Pipeline
Pod
K8s Deployment(s) ▸ Max 110 pods per
node
▸ Max 5,000 nodes per
cluster
▸ Max 150,000 pods in
total
▸ *Separate Kafka and
Kafka Connect
clusters

8. State-of-the-art Orchestration
8
PROCESS ORCHESTRATION
▸ We started out on a single VM and moved to a distributed
process orchestration tool
▸ Kafka’s ecosystem is lagging state of the art process
orchestration like Kubernetes, Nomad, etc.
▸ ksqlDB and Kafka Connect manage processes, but we will
see how they are lacking fundamental patterns to be
operated at scale
Kafka Streams
Single VM Docker
Java Quarkus
Kubernetes

9. Quarkus SmallRye Reactive Messaging
9
STARTUP TIME
Scheduled
0s
Scheduled
60s
First Event
Processed
First Event
Processed
5s …
Liveness
OK
10s
Kafka Streams
30s

10. 10
WORKLOAD DENSITY
Docker on Single VM
Quarkus
Quarkus
Quarkus
Kubernetes
Kafka Streams
Kafka Streams
RAM < 1.5GB
RAM < 1.5GB
RAM < 300MB
RAM < 300MB
RAM < 300MB

11. 11
STRIMZI FOR KAFKA
…
Kubernetes; Dedicated node pool for Kafka
Apache Kafka
Broker
Apache Kafka
Broker
Apache Kafka
Broker
Strimzi Kafka
Operator
Kubernetes
StatefulSet
Pod Pod Pod

12. 12
CONSUMER RE-BALANCING
summit
consumer 0
summit
consumer 1
summit
consumer 2
…
Pod Pod Pod
summit
partition 0
summit
partition 1
summit
partition 2
… … …
…
… …

13. 13
CONSUMER RE-BALANCING
summit
consumer 0
summit
consumer 1
summit
consumer 2
…
Pod Pod Pod
summit
partition 0
summit
partition 1
summit
partition 2
… … …
…
… …
▸ Consumer re-balancing will
cause no consumption until
re-balancing is completed
by co-ordinator
▸ Number of consumers can
change due to errors,
disconnection, and
triggered by new load
requirements

14. 14
UNEXPECTED SHUTDOWN
Startup Time
Partition Size

15. 15
UNEXPECTED SHUTDOWN
Startup Time
Partition Size
Not Linear
Point of No Recovery
Log Size

16. How to deal with resource-leaking Kafka
Connect connectors?
16

17. 17
KAFKA CONNECT SELF-MANAGED
…
Kubernetes (K8s); Dedicated Kafka Connect Nodepool
ElasticSearch Sink
Task C
S3 Source
Task A
PostgreSQL Source
Task A
K8s Deployment / Connect Cluster
Pod Pod Pod
PostgreSQL Source
Task B
MySQL CDC Source
Task A
MySQL CDC Source
Task B

18. K8s Deployment / Connect Cluster
18
KAFKA CONNECT SELF-MANAGED
…
Kubernetes (K8s); Dedicated Kafka Connect Nodepool
ElasticSearch Sink
Task C
MySQL CDC Source
Task A
MySQL CDC Source
Task B
S3 SOURCE
TASK A
Pod Pod
PostgreSQL Source
Task A
PostgreSQL Source
Task B

19. K8s Deployment / Connect Cluster
19
KAFKA CONNECT SELF-MANAGED
…
Kubernetes (K8s); Dedicated Kafka Connect Nodepool
ElasticSearch Sink
Task C
S3 SOURCE
TASK A
Pod Pod
MySQL CDC Source
Task A
MySQL CDC Source
Task B
PostgreSQL Source
Task A
PostgreSQL Source
Task B

20. K8s Deployment / Connect Cluster
20
KAFKA CONNECT SELF-MANAGED
…
Kubernetes (K8s); Dedicated Kafka Connect Nodepool
ELASTICSEARCH SINK
TASK C
S3 SOURCE
TASK A
Pod Pod
MySQL CDC Source
Task A
MySQL CDC Source
Task B
PostgreSQL Source
Task A
PostgreSQL Source
Task B

21. Connect Cluster Connect Cluster
21
KAFKA CONNECT SELF-MANAGED
…
Kubernetes (K8s); Dedicated Kafka Connect Nodepool
ElasticSearch Sink
Task C
S3 Source
Task A
Connect Cluster
Pod Pod Pod
MySQL CDC Source
Task A
MySQL CDC Source
Task B
PostgreSQL Source
Task A
PostgreSQL Source
Task B

22. Connect Cluster Connect Cluster
22
KAFKA CONNECT SELF-MANAGED
…
Kubernetes (K8s); Dedicated Kafka Connect Nodepool
ElasticSearch Sink
Task A
PostgreSQL Source
Task A
Connect Cluster
Pod Pod
S3 Source
Task A
Pod
MySQL CDC Source
Task A

23. Connect Cluster Connect Cluster
23
KAFKA CONNECT SELF-MANAGED
…
Kubernetes (K8s); Dedicated Kafka Connect Nodepool
S3 SOURCE
TASK A
ElasticSearch Sink
Task A
PostgreSQL Source
Task A
Connect Cluster
Pod Pod
MySQL CDC Source
Task A

24. Connect Cluster Connect Cluster
24
KAFKA CONNECT SELF-MANAGED
…
Kubernetes (K8s); Dedicated Kafka Connect Nodepool
S3 SOURCE
TASK A
ElasticSearch Sink
Task A
PostgreSQL Source
Task A
Connect Cluster
Pod Pod
MySQL CDC Source
Task A

25. Connect Cluster Connect Cluster
25
KAFKA CONNECT SELF-MANAGED
…
Kubernetes (K8s); Dedicated Kafka Connect Nodepool
ElasticSearch Sink
Task A
PostgreSQL Source
Task A
Connect Cluster
Pod Pod
S3 Source
Task A
Pod
MySQL CDC Source
Task A

26. ▸ Utilise state of the art orchestration tools.
▸ Running Kafka on Kubernetes does not bring automatic elasticity.
▸ Kafka Connect is not self-contained. This will become a larger headache the more
connector tasks are running in a given cluster.
▸ Think about startup time throughout your tech stack. From Kafka brokers over Connect
tasks to streaming applications.
26
TAKE-AWAYS
Key Learnings

27. How to monitor and debug pipelines?
27

28. 28
MONITORING STREAMING DATA PIPELINES
Kafka Topic
Kafka
Connect
Source
Connector
Kafka
Connect
Sink
Connector
Kafka Topic Kafka Streams

29. ▸ External data sources or data sinks are unavailable (temporarily)
▸ Consumers (processors or sink connectors) are slower than producers
▸ Processing of events fails
29
POTENTIAL PRODUCTION ISSUES
Most common issues in streaming data pipelines

30. ▸ External data sources or data sinks are unavailable (temporarily)
▸ Consumers (processors or sink connectors) are slower than producers
▸ Processing of events fails
30
POTENTIAL PRODUCTION ISSUES
Most common issues in streaming data pipelines

31. 31
MONITORING CONNECTORS
Kafka Topic
Kafka
Connect
Source
Connector
Kafka
Connect
Sink
Connector
Kafka Topic Kafka Streams
Monitoring the health of connectors
‣Periodically call /connectors/:connector_name/status and investigate the response

32. 32
MONITORING CONNECTORS
GET /connectors/hdfs-sink/status
{
"name": "hdfs-sink",
"connector": {
"state": "RUNNING",
"worker_id": "localhost:8083"
},
"tasks":
[
{
"id": 0,
"state": "RUNNING",
"worker_id": “localhost:8083"
}
]
}
Healthy

33. 33
MONITORING CONNECTORS
GET /connectors/hdfs-sink/status
{
"name": "hdfs-sink",
"connector": {
"state": “FAILED",
"worker_id": "localhost:8083"
},
"tasks":
[
{
"id": 0,
"state": "FAILED",
"worker_id": “localhost:8083”,
"trace": "org.apache.kafka.common.errors.RecordTooLargeExceptionn"
}
]
}
Unhealthy

34. 34
MONITORING CONNECTORS
Kafka Topic
Kafka
Connect
Source
Connector
Kafka
Connect
Sink
Connector
Kafka Topic Kafka Streams
‣Periodically call /connectors/:connector_name/status and investigate the response
‣If failed, try to restart the connector (e.g., deals with temporary API outages) and
escalate or alert after X restarts
‣Sometimes, directly escalating might be reasonable
Monitoring the health of connectors

35. ▸ External data sources or data sinks are unavailable (temporarily)
▸ Consumers (processors or sink connectors) are slower than producers
▸ Processing of events fails
35
POTENTIAL PRODUCTION ISSUES
Most common issues in streaming data pipelines

36. 36
MONITORING BACKPRESSURE
Consumer Lags
Kafka Topic Consumer
‣Difference between the latest offset available in the Kafka topic (partition) and the
latest offset processed by the consumer
‣Resembles how much consumers are behind producers in terms of number of records
processed

37. 37
MONITORING BACKPRESSURE
Kafka Topic
Kafka
Connect
Source
Connector
Kafka
Connect
Sink
Connector
Kafka Topic Kafka Streams
Consumer Lags in Streaming Data Pipelines

38. 38
MONITORING BACKPRESSURE
Kafka Topic
Kafka
Connect
Source
Connector
Kafka
Connect
Sink
Connector
Kafka Topic Kafka Streams
Kafka Streams Consumer Lag
‣Number of records that have been extracted by the data source connector but have not
yet been processed by the Kafka Streams app
‣If data processing is slower than extraction, you might want to increase the degree of
parallelism of the Kafka Streams app

39. 39
MONITORING BACKPRESSURE
Kafka Topic
Kafka
Connect
Source
Connector
Kafka
Connect
Sink
Connector
Kafka Topic Kafka Streams
Sink Connector Consumer Lag
‣Number of records that have been processed by the Kafka Streams app but have not yet
been published by the sink connector
‣If publishing data to the data sinks is slower than processing, you might want to increase
the number of tasks of the sink connector

40. ▸ External data sources or data sinks are unavailable (temporarily)
▸ Consumers (processors or sink connectors) are slower than producers
▸ Processing of events fails
40
POTENTIAL PRODUCTION ISSUES
Most common issues in streaming data pipelines

41. 41
DEAD-LETTER QUEUES
Keep track of errors in processing
‣By default, Kafka Connect connectors fail
when observing errors in processing
‣We recommend to configure a dead-letter
queue (topic) for storing records that could
not be processed
‣Monitor the dead-letter queue topic and
manually investigate failed records
errors.tolerance = all
errors.deadletterqueue.topic.name = topic-dlq
Topic
Dead-letter
queue topic
Successful
processing
Failed
processing
Kafka
Connect
Source
Connector

42. What are ways to deal with large data
sources or slow data sinks?
42

43. 43
DEALING WITH LARGE DATA SOURCES
‣Hurts a lot when performing initial snapshots,
which can take hours
‣Use multiple connectors for the same database
and make use of table.include.list
‣Adjust the snapshot query and consider only a
subset of the data source
‣Mitigate pain with incremental snapshotting
‣Accelerate snapshotting with parallelisation
PostgreSQL
Debezium
Source
Connector
TBs of data

44. 44
DEALING WITH SLOW DATA SINKS
Kafka
Connect
Sink
Connector
Elasticsearch
‣Detect slow data sinks by monitoring the sink
connector consumer lag
‣Parallelise sending records to the data sink by
increasing the number of connector tasks
‣If available, batch multiple records and send
them with one request to the data sink
‣Avoid duplicated data delivery by adjusting
max.poll.records or max.poll.interval.ms

45. What is missing in today’s ecosystem for
streaming to become a commodity?
45

46. 46
SERVERLESS TOPICS
‣Partitioned topics are the de-facto standard for
persisting events
‣# partitions = maximum degree of parallelism
‣Choosing the number of partitions remains a crucial
questions with significant impact on future cost and
performance, and needs to be answered at topic
creation time (!)
‣Having the ability to dynamically choose the degree of
parallelism would allow to easier cope with peak loads
"Horizontal Partition Autoscaler”
Partition 0
1 partition
Partition 0 Partition 1 Partition 2
3 partitions
Partition 0
1 partition
Scale Up
Scale Down

47. 47
EASE OPERATIONS
More and better managed services
‣Operating streaming data pipelines boils down to running multiple distributed
systems and remains one of the big hurdles for its adoption
‣Managed services can reduce the operational pain
‣We witness the rise of cloud/SaaS offerings but believe there is still lots of room for
improvement

48. Summary
48

49. 49
TAKE-AWAYS
Summary
‣Throwing Kafka and Kafka Connect at Kubernetes is beneficial but does not provide
a true cloud-native experience. It takes a few steps to, for instance, apply the self-
containment principle to Kafka Connect.
‣If possible, try to handle errors of connectors or streaming applications in an
automated manner without bringing the pipeline down
‣A lot of issues occur when integrating external systems that you do not control, e.g.,
snapshotting a very large database table, sending events to slow APIs, etc.

50. Questions?
50