This document compares the ingestion performance of Kafka and Kinesis when using multiple producers on a single node. It finds that Kafka can ingest data at a higher throughput than Kinesis, and that throughput increases with more partitions and producers. There were some challenges in automating the installation and setup. The document also provides background on the author and costs of running the experiments on AWS.

1.
Ingestion
Comparison
Thomas
Schreiter
Insight
Data
Engineering
Fellow

2. Ingestion
=
Message
Queuing
System
ProducerProducerProducers
ProducerProducerConsumers

3. Research
Question:
How
fast
can
data
be
produced
into
Kinesis/Kafka
if
all
producers
run
on
only
one
node?
ProducerProducerProducers
1x
m3.medium

4. DEMO
….

5. Throughput
over
#producers
5

6. 6
0"
5000"
10000"
15000"
20000"
25000"
30000"
35000"
1" 2" 5" 10" 20" 50" 100" 200" 500"
Throughput)[msg/sec])
Bulk)Size)[msg])
Throughput)over)Bulk)Size)
Ka)a"
Kinesis"

7. ProducerProducerProducer.py
ProducerProducerProducer.py
4x
m3.large
1x
m3.medium
1x
m3.medium
1
stream
“Message #0 to Kafka @ 12:39:04.300”
“Message #1 to Kafka @ 12:39:04.310”
…
“Message #0 to Kinesis @ 13:00:05.700”
“Message #1 to Kinesis @ 13:00:05.702”
…

8. logger
metrics
ProducerProducerProducer.py
ProducerProducerProducer.py
4x
m3.large
1x
m3.medium
1x
m3.medium
1
stream
“Message #0 to Kafka @ 12:39:04.300”
“Message #1 to Kafka @ 12:39:04.310”
…
“Message #0 to Kinesis @ 13:00:05.700”
“Message #1 to Kinesis @ 13:00:05.702”
…

9. logger
metrics
“Message #0 to Kafka @ 12:39:04.300”
“Message #1 to Kafka @ 12:39:04.310”
…
“Message #0 to Kinesis @ 13:00:05.700”
“Message #1 to Kinesis @ 13:00:05.702”
…
ProducerProducerProducer.py
ProducerProducerProducer.py
4x
m3.large
1x
m3.medium
1x
m3.medium
1x
m3.medium
1x
t2.micro
1
stream

10. Engineering
Challenges
Install
scripts:
tried
to
automate
everything
☺

11. Engineering
Challenges
Install
scripts:
tried
to
automate
everything
☺
broke
Kafka
installation
in
Week
2
☹

12. Engineering
Challenges
Install
scripts:
tried
to
automate
everything
☺
broke
Kafka
installation
in
Week
2
☹
and
again
in
Week
4
☹ ☹ ☹

13. but
Engineering
puzzles
are
really
fun
☺☺☺

14. And
I
read
Kafka
for
the
first
time

15. Thomas
Schreiter
[thomas.dataengineer@gmail.com]
M.Sc.
+
B.Sc.
in
Computer
Science
@Karlsruhe
Institute
of
Technology,
Germany
Ph.D.
in
Transportation
@Delft
University
of
Technology,
The
Netherlands
Before
Insight:
Research
Engineer
in
Transportation
@UC
Berkeley

17. AWS
Costs
17

19. Throughput
over
#partitions
Throughput
[#msg/sec]
0
300
600
900
1200
1
par__on 2
par__ons 3
par__ons 4
par__ons
Ka`a
Kinesis

20. Older
results
Throughput
[#msg/sec]
0
500
1000
1500
2000
1
par__on 2
par__ons 3
par__ons 4
par__ons
Ka`a
Kinesis