An introduction to the kafka stream processing platform. The presentation gives a small introduction into stream processing and furthermore explains how kafka streams and kafka connect are used together to implement realtime stream processing flows.

1. Apache Kafka
Stream processing made easy

3. Streams 101
An introduction to stream
processing

4. Transformation of
a stream of data fragments
into a continuous flow of
information

5. Stream Processing
Get real-time insights
Lower processing latency
Easier to test
Easier to maintain
Easier to scale
Different way of thinking
At-least-once vs exactly-
once
Time
+ -

6. Every company is already
doing stream processing
(more or less ... )

7. A Stream
Key 1 -> value 1
Key 2 -> value 2
Key 1 -> value 3
...

8. A Table
+-------+---------+
| Key 1 | value 3 |
| Key 2 | value 2 |
+-------+---------+

9. A Table through time
+-------+---------+
| Key 1 | value 1 |
+-------+---------+
Timestamp 1
+-------+---------+
| Key 1 | value 1 |
| Key 2 | value 2 |
+-------+---------+
Timestamp 2
+-------+---------+
| Key 1 | value 3 |
| Key 2 | value 2 |
+-------+---------+
Timestamp 3
Timestamp ...

10. Let’s remove the redundancy

11. A Table through time as SETs
SET(key1 -> value1)
Timestamp 1
SET(key2 -> value2)
Timestamp 2
SET(key 1 -> value3)
Timestamp 3
Timestamp ...

12. SET(key1 -> value1)
SET(key2 -> value2)
SET(key1 -> value3)
Changelog
key1 -> value1
key2 -> value2
key1 -> value3
Stream

13. Tables are
materialized views
of streams

14. Why is this important?

15. Events used to manipulate core data.
Today events are our core data
Daan Gerits, 2012

16. Every stream process app is a
combination of state and streams

17. Streaming vs batch
is like
agile vs waterfall
but then for data.

18. Kafka
A Stream Processing Platform

19. Kafka
Proxy
Kafka
Streams
Kafka
Connect
Kafka
Security
Schema
Repo
Kafka

20. Kafka Platform
Streams and Connect apps are just (java) apps
Streams and Connect are libraries
Can be deployed like any other (java) app
Multiple instances of the same app can be launched
Use tools like Mesos, kubernetes, Docker Swarm, ...

21. Batch Microbatch
Flink
Kafka
Spark
Storm / Heron
Event

22. Build apps, not Jobs

23. Kafka
Proxy
Kafka
Streams
Kafka
Connect
Kafka
Security
Schema
Repo
Kafka

24. Kafka Engine
A message broker with a twist

25. Kafka Engine
Producer ConsumerTopic
message message
Producer
Producer Consumer
Consumer

26. Kafka Engine
Producer ConsumerTopic
message message
Producer
Producer Consumer
Consumer

27. Messages
Contain byte arrays
Have a
Timestamp
Key
Value

28. Topics
Are more like datastores
Uses disk instead of memory
Retains the messages
Are partitioned and replicated

29. Wait?? … Disk??

30. Sequential disk access is fast*
* Don’t believe me? Read http://kafka.apache.org/documentation#persistence

31. Producer
Puts messages onto kafka
Determines the partition to write to
Can be implemented in many, many languages

32. Consumer
Gets messages from kafka
Can be grouped into Consumer Groups
Allows for round robin message delivery
Enables scaling of consumers
Have a persisted offset per Consumer Group
Stored in Zookeeper
Or in Kafka

33. Kafka Engine
Producer Consumer
Topic
Partition B
Producer Consumer
Topic
Partition A

34. 100 000 msg/sec
On a barely tweaked, 3 node cluster

35. 2 000 000 msg/sec
On a heavily tweaked cluster
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

36. Kafka Connect
Getting data in and out

37. A
Simple and scalable
way to get
data in and out
of topics

38. Kafka Connect
Datasource Topic
Kafka
Connect
Topic Datasink
Kafka
Connect
Or

39. Kafka Connect
Datasource Topic
Kafka
Connect
Kafka
Connect
Kafka
Connect

40. Kafka Connect
MySQL ⬢ Salesforce ⬢ Redis ⬢ MQTT ⬢
InfluxDB ⬢ RethinkDB ⬢ HBase ⬢ Solr ⬢
Couchbase ⬢ Elasticsearch ⬢ Hazelcast ⬢
Google PubSub ⬢ HDFS ⬢ S3 ⬢ Splunk ⬢
Spooldir ⬢ JDBC ⬢ Syslog ⬢ Cassandra ⬢
Vertica ⬢ DB2 ⬢ Goldengate ⬢ Jenkins ⬢
PredictionIO ⬢ JMS ⬢ Twitter ⬢ Attunity ⬢
MSSQL ⬢ Postgres ⬢ DynamoDB ⬢ IRC ⬢
Kudu ⬢ Ignite ⬢ MongoDB ⬢ Bloomberg
Ticker ⬢ FTP

41. Kafka Streams
Processing streaming data

42. Kafka Streams
Topic Topic
Kafka
Streams
Topic
Topic

43. Kafka Streams
KStream for a stream of data
KTable to keep the latest value for each key
KTable state is distributed across app instances
Transform from streams to tables and tables to streams
Choose which field to use as “timestamp”

44. TOPIC A
TOPIC B
TOPIC C
Kafka
Connect
App
Kafka
Streams
App
Kafka
Streams
App
Kafka
Connect
App
TOPIC C
TOPIC B
TOPIC A

45. So how do you build
solutions with this?

46. Kafka
Kafka
Connect
Kafka
Streams
Kafka
Kafka
Kafka
Connect

47. TOPIC A
TOPIC B
TOPIC C
Sales JDBC
Kafka
Connect
Top
Products
Ranker
Emailer
TOPIC C
TOPIC B
TOPIC A
Low Stock
Notifier
Kafka
Connect
App
Slack Poster

48. Proposal