Kafka est devenu en quelques temps un outil central dans les architectures d'analyses de données. Ce journal distribué repose sur quelques principes simples qui en font un outil performant et robuste et lui offre une scalabilité presque linéaire. Nous vous proposons de comprendre par la pratique quelques concepts techniques en codant un système de production/consommation de messages. Matériel nécessaire : un ordinateur (Machine ou VM Unix uniquement car Kafka ne fonctionne pas sous Windows) sur lequel vous aurez préalablement installé votre IDE préféré (IntelliJ/ Eclipse) ainsi que Maven ou SBT. De plus, il vous faut cloner le git clone https://github.com/xebia-france/kafka_the_north_face

1. Ka ka par la face Nord

2. Messaging System?

3. Jay Kreps
Neha Narkhede
Jun Rao
History

4. WebAppRelational
DB
NoSQL
DB
DWH
Hadoop
Monitoring Logs

5. WebApp
Relational
DB
NoSQL
DB
DWH
Hadoop
ActiveMQ
WebApp
Logs
Monitoring
WebApp
Search

6. WebApp
Relational
DB
NoSQL
DB
DWH
Hadoop
ActiveMQ
WebApp
Logs
Monitoring
WebApp
Search
BIG
MESS

7. Stream Data Platform

8. ● Distributed
● High throughput
● Large number of consumers
● Ad-hoc consumers
● Batch consumers
● Automatic recovery from broker failure
Features

9. Distributed Commit Logs
10 11 12 13 14 15 16 17 18987654321 19
1st record
Next record
Written

10. Consumer
Broker
Consumer
Consumer
Kafka Cluster
Broker Broker
Broker Broker Broker
Zookeeper
Producer
Producer
Producer
Architecture

11. Producer
10 11 12 13 14 15 16 17 18987654321
10 11 12 13 14987654321 15
10 11 12 13 14 15987654321 16
Partition #1
Partition #2
Partition #3
ProducerProducer
19
16
17
offset
Old New
Writes
Writes
Writes

12. Consumer group

13. Consumer group
10 11 12 13 14 15 16 17 18987654321
10 11 12 13 14987654321 15
10 11 12 13 14 15987654321 16
Partition #1
Partition #2
Partition #3
Group Topic # Offset
1 log 1 18
1 log 2 12
1 log 3 14
2 log 1 1
2 log 2 0
2 log 3 3
Consumer group 1 Consumer group 2
Old New

14. Topic storage
10 11 12 13 14 15 16 17 18987654321
Partition #1
directory segment = file
app_log-2:
total 1864
-rw-r--r-- 1 root root 512 Oct 28 01:00 00000000000000208027.index
-rw-r--r-- 1 root root 337762 Oct 27 19:03 00000000000000208027.log
-rw-r--r-- 1 root root 10485760 Oct 28 19:24 00000000000000208739.index
-rw-r--r-- 1 root root 1553051 Oct 28 19:24 00000000000000208739.log
app_log-3:
total 1940
-rw-r--r-- 1 root root 48 Oct 27 07:02 00000000000000207555.index
-rw-r--r-- 1 root root 31360 Oct 27 04:05 00000000000000207555.log

15. Topic clustering
10 11 12 13 14 15 16 17 18987654321
10 11 12 13 14987654321 15
10 11 12 13 14 15987654321 16
Partition #1
Partition #2
Partition #3
Leader

16. Topic clustering
10 11 12 13 14 15 16 17 18987654321
10 11 12 13 14987654321 15
10 11 12 13 14 15987654321 16
Partition #1
Partition #2
Partition #3
Leader Replica Replica

17. Topic clustering
10 11 12 13 14 15 16 17 18987654321
10 11 12 13 14987654321 15
10 11 12 13 14 15987654321 16
Partition #1
Partition #2
Partition #3

18. Jay Kreps
Neha Narkhede
Jun Rao
Ka ka Enterprise Ready
2011 2012
2014

19. ● User behaviour, click stream analysis
● Infrastructure monitoring and security
● Telemetry data from mobile/sensors
● IoT
● Log analysis
● ...
Usage

20. Used by
● LinkedIn : activity stream, metrics
● Netflix : Real-time Monitoring
● Twitter : Real-time data pipeline
● Spotify : log delivery
● Loggly : log collection and processing
● Mozilla : telemetry data
● Airbnb, Square, Uber, Criteo, OVH ...

21. ● Need to write code to use it (no ready
made producers and consumers)
● Not JMS replacement
● No data transformations yet
● No encryption, authorization or
authentication yet
(v0.9.0 KAFKA-2210, KAFKA-2211)
Pain Points

22. Hand’s On : The Road
Installation
Zookeeper
Brokers
Topic
Console Tools

23. Hand’s On : The Trail
Producer Kafka
(Java/Scala)
High Level Consumer
(Java/Scala)

24. Hand’s On : The North Face
“Simple” Consumer

25. Go!