Kafka is a real-time, fault-tolerant, scalable messaging system. It is a publish-subscribe system that connects various applications with the help of messages - producers and consumers of information.

1. Apache Kafka - Messaging System
Dmitry Tolpeko, EPAM Systems – September 2014

2. Kafka Overview
Kafka is a real-time, fault-tolerant, scalable messaging
system.
2
It is a publish-subscribe system that connects various
applications with the help of messages - producers
and consumers of information.
Producers and consumers are independent,
messages are queued, one producer can serve
multiple consumers.
Was originally developed by LinkedIn.

3. SECTION
Apache Kafka
CONCEPTS
3

4. Kafka Architecture
4
Client Server Client
Producer(s) Broker(s) Consumers(s)
ZooKeeper
• Brokers act as the server part of Kafka. Brokers are peers,
there is no the master broker.
• Brokers can run on multiple nodes, but you can also run
multiple brokers on each node. Each broker has own IP and
port for client connections.

5. Topic is a way to handle
multiple data streams
(different data feeds i.e.)
Each producer sends
messages to, and
consumers read the
messages from the
specified topic.
New topics can be
created automatically
when a message with a
new topic arrives, or you
can use --create
command to create a
topic.
Topics
5
Broker
Topic 1
Topic 2
Producer 1
Producer 2
Producer 3
Consumer 1
Consumer 2

6. A topic can contain one or more partitions.
Each partition is stored on a single server, and
multiple partitions allow the queue to scale
and go beyond the limits of a single system.
Partitions also allow a single consumer to
concurrently read messages in multiple
concurrent threads. You can add new
partitions dynamically.
Offset is uniquely identifies a message within
partition.
Partitions
6
Broker 1
Topic 1
Partition 1
Topic 2
Partition 1
Broker 2
Topic 2
Partition 2
Partition 3

7. Each partition is replicated for fault-tolerance.
Partition has one server that acts a Leader, it
handles all read-write requests.
Zero or more servers act as Followers, they
replicate the leader and if it fails one of them
becomes the new Leader.
Leader uses ZooKeeper heartbeat mechanism to
indicate that it is alive.
A follower acts as a normal consumer, it pulls
messages and updates own log. Only when all
followers (ISR group) sync the message it can be
send to consumers. When a follower rejoins after
a downtime it can re-sync.
Replication
7
Broker 1
Topic 1
Partition 1 - Leader
Broker 2
Topic 1
Partition 1 - Follower

8. Consumers are organized to consumer
groups.
To consume a single message by multiple
consumers, they must belong to different
consumer groups.
A consumer group is a single consumer
abstraction, so consumers from single group
read messages like from a queue there is no
message broadcast within the group. This
helps balance load among consumers of the
same type (fault-tolerance, scalability).
The state of consumed messages are
handled by consumers, not brokers.
Consumers store the state in ZooKeeper -
offset within each partition for each
Consumer group, not consumer (!)
Consumer group name is unique within the
Kafka cluster.
Consumer Groups
8
Topic 1
Partition 1
Partition 2
Partition 3
Group 1
Consumer
Consumer
Group 2
Consumer

9. Order Guarantees and Delivery Semantics
Each partition can be consumed only one consumer within the consumer group.
Kafka only provides total order guarantee within a partition, not between different
partitions in a topic.
If you need total order over messages you have to use one partition, and in this
case you can use only one consumer process.
Kafka guarantees at-least-once delivery semantics by default where
messages are never lost but may be redelivered (keys can be used to handle
duplicates). Kafka offers options to disable retries (so messages can be lost) in
case if the application can handle this, and needs a higher performance.
Kafka retains all published messages - no matter whether they are consumed or
not - for the configured period of time (2 days by default).
9

10. Producer can assign a key for a message
that defines which partition to publish
message to.
• Random (default, when no partition
class or key specified)
• Round-robin for load balancing
• Partition function (hash by message
key i.e.) - if key is a class type
(Source ID i.e.) then all messages of
the same type go to one partition.
Producer can optionally require an
acknowledgment from the broker that the
message was received (synced to Leader
or all followers).
Kafka can group multiple messages and
compress them.
Producers
10
Producer 1
Topic 1
Partition 1
Partition 2
Partition 3
Producer 2

11. Consumers read the messages from the
brokers leading the partitions (pull method).
A consumer labels itself with a consumer
group.
If the number of consumers of a specific
consumer group is greater than the number
of partitions, then some consumers will never
see a message.
If there are more partitions than consumers
of a specific consumer group, then a
consumer can get messages from multiple
partitions (no order guarantee). Then when
you add consumers, Kafka re-balances
partitions.
Consumers can get compressed message as
a single message.
Consumers
11
Partition 1
Partition 2
Partition 3
Group 1
Consumer 1
Consumer 2
Consumer 3
Consumer 4

12. Consumer Advanced Features
There are High Level and Simple Consumer API.
A High Level Consumer sets
auto.commit.interval.ms option that defines how often
offset is updated in ZooKeeper. If an error occurs between
updates, the consumer will get replayed messages (!)
Simple Consumer is a low-level API that allows you to set
any offset, explicitly read messages multiple times, or ensure
that a message is processed only once.
12

13. SECTION
Apache Kafka
INTERNALS
13

14. Kafka relies heavily on OS disk cache, not
JVM heap even for caching messages. Data
immediately written (appended) to a file.
Consumed messages are not deleted.
Data files (called logs) are stored at
log.dirs
A directory exists for each topic partition that
contains log segments (files 0000000.log -
named as offset of the 1st message in the
log). log.segment.bytes and
log.roll.hours define rotation policy.
log.flush.interval.xxx options define
how often fsync performed on files.
All options can be specified either globally or
per topic.
Persistence
14
Broker JVM App
OS page cache
/data/kafka-logs
TopicName-0
00000.log

15. Messages can be grouped
together to minimize the number of
network round-trips.
Multiple messages can be also
compressed together (GZIP,
Snappy) that helps achieve a good
compression rate and reduce
amount of data sent over network.
Producer can specify
compression.codec and
compressed.topics
Network I/O
15
Message1
Message2
Message3
Compressed
Network

16. There is no in-memory application
level cache, data are in the OS
pagecache.
Kafka uses sendfile Linux API
calls that directly sends data from
pagecache to a network socket, so
there is no need to do read/write
to application memory space.
Grouped messages are stored
compressed in the log, and
decompressed only by consumers.
Memory
16
Broker JVM App
OS page cache
Network

17. Log Compaction
Without log compaction (time series data):
17
Key1 Key2 Key3 Key1 Key2 Key1 Key3
A B C AA BB AAA CC
With log compaction only the last update is stored for each key:
Key2 Key1 Key3
BB AAA CC
Log compaction can be defined per topic. This can help increase
performance of roll-forward operations, and reduce storage.

18. Kafka Use Cases
• Messaging - decouple processing or handle message
buffer
• Monitoring and Tracking - collect activity, clickstream,
status data and logs from various systems
• Stream Processing - aggregate, enrich, handle micro-batches
etc.
• Commit Log - facilitate replication between systems
18

19. Thanks!
Join us at
https://www.linkedin.com/groups/Belarus-
Hadoop-User-Group-BHUG-8104884
dmitry_tolpeko@epam.com