This document discusses using microservices with Kafka. It describes how Kafka can be used to connect microservices for asynchronous communication. It outlines various features of Kafka like high throughput, replication, partitioning, and how it can provide reliability. Examples are given of how microservices could use Kafka for logging, filtering messages, and dispatching to different topics. Performance benefits of Kafka are highlighted like scalability and ability to handle high volumes of messages.

1. Microservices with Kafka
László-Róbert ALBERT
Dan BALESCU Casablanca • 02 November 2016

2. Róbert
Dan

3. ING Software Development Center
Bucharest
3

4. ING Software Development Center
Bucharest
4
IT
5,7% of GDP with 100k ppl
Agriculture
4,4% of GDP with 1.8 million ppl

5. 5
FAULT
TOLERANCE
PERFORMANCE
RELIABILITY
SECURITY
AVAILABILITY
HIGH
SCALABILITY
REPLICATION
CHANGE
SELF HEALING

6. 6

7. 7

8. 8
Account
Manager

9. 9
Account
Manager
Core
Banking

10. 10
Account
Manager
Core
Banking
Transfer

11. 11
Account
Manager
Core
Banking
Transfer

12. 12
Source: http://www.slideshare.net/dberkholz/
how-microservices-are-redefining-
modern-application-architecture

13. 13
Source: http://www.slideshare.net/dberkholz/
how-microservices-are-redefining-
modern-application-architecture

14. 14

15. 500 million messages / day
15

16. 500 million messages / day 700 billion messages / day
16

17. 500 million messages / day 700 billion messages / day 1.4 trillion messages / day
17

18. 500 million messages / day 700 billion messages / day 1.4 trillion messages / day
18

19. Messaging Brokers Features
19
Kafka ActiveMQ RabbitMQ

20. Messaging Brokers Features
20
Support for multiple protocols
- Advanced Message Queueing Protocol
- Streaming Text Oriented Messaging Protocol
- Message Queuing Telemetry Transport
- Etc
Kafka ActiveMQ RabbitMQ

21. Messaging Brokers Features
21
Support for multiple protocols
- Advanced Message Queueing Protocol
- Streaming Text Oriented Messaging Protocol
- Message Queuing Telemetry Transport
- Etc
Kafka ActiveMQ RabbitMQ
Java Messaging System

22. Messaging Brokers Features
22
Support for multiple protocols
- Advanced Message Queueing Protocol
- Streaming Text Oriented Messaging Protocol
- Message Queuing Telemetry Transport
- Etc
Kafka ActiveMQ RabbitMQ
Java Messaging System
Java Authentication and Authorization Service
Simple Authentication and Security Layer

23. Messaging Brokers Features
23
Support for multiple protocols
- Advanced Message Queueing Protocol
- Streaming Text Oriented Messaging Protocol
- Message Queuing Telemetry Transport
- Etc
Kafka ActiveMQ RabbitMQ
Java Messaging System
Java Authentication and Authorization Service
Simple Authentication and Security Layer
Transactions

24. Messaging Brokers Features
24
Support for multiple protocols
- Advanced Message Queueing Protocol
- Streaming Text Oriented Messaging Protocol
- Message Queuing Telemetry Transport
- Etc
Kafka ActiveMQ RabbitMQ
Java Messaging System
Java Authentication and Authorization Service
Simple Authentication and Security Layer
Transactions
Filtering

25. Messaging Brokers Features
25
Support for multiple protocols
- Advanced Message Queueing Protocol
- Streaming Text Oriented Messaging Protocol
- Message Queuing Telemetry Transport
- Etc
Kafka ActiveMQ RabbitMQ
Java Messaging System
Java Authentication and Authorization Service
Simple Authentication and Security Layer
Transactions
Filtering
Persistence

26. 26
Performance
Features

27. 27
Messaging Brokers Performance
Message producing Message consumption
LinkedIn – Kafka: a Distributed Messaging System for Log Processing

28. 28
Performance
Features

29. 29
Usecases of Kafka
Consumer
Microservice A Microservice B
Transport API
Producer

30. 30
Usecases of Kafka
Consumer
Microservice A Microservice B
Transport API
Producer
Consumer
Producer

31. 31
Usecases of Kafka
Microservice A Microservice B
Transport API
Producer
Consumer
Microservice C
Consumer

32. 32
System Availability & Performance

33. 33
System Availability & Performance

34. 34
Scalability

35. 35
Scalability

36. 36
Usecases of Kafka
Microservice A ELK stackKafka ConsumerKafka Appender
K
A
F
K
A

37. © ING 2016
37
Producer
// build a 'default' factory based on your-implicit-kafka-producer.properties
// - the properties file contains the list of brokers to connect to
ProducerFactory factory = ProducerFactory.builder().build();
// create a message producer with a Payload that sends the messages to topic ‘test’
Producer<Message<Payload>> producer =
factory.createMessageProducer("test", Payload.class);
// send a Message { header: Map<String, String>; payload: Payload }
producer.send(
Message.<Payload> builder()
.header("applicationId", "sample-producer")
.header("timestamp", now())
.payload(new Payload("/some/file/path"))
.build()
);

38. 38
Producer
Native
Kafka Producer
API
your-implicit-producer.properties
your-security.properties

39. 39
Consumer
// build a 'default' factory – your-implicit-kafka-consumer.properties
// - the properties contains the list of brokers from which it will receive messages
ConsumerFactory factory = ConsumerFactory.builder().build();
// obtain a consumer instance that can process Message<Payload> messages
Consumer<Message<Payload>> consumer = factory.createMessageConsumer(Payload.class);
// prepare the message (Message { header: Map<String, String>; payload: Payload } ) handler
ReceiverHandler<Message<Payload>> messageReceiver =
new ReceiverHandler<Message<Payload>>() {
@Override
public void onReceive(Message<Payload> message) {
System.out.printf("message received: %s%n", message);
}
};
// register the handler to receive messages from the topic named “test”
consumer.registerHandler("test", messageReceiver);
© ING 2016

40. 40
Consumer
Native
Kafka Consumer
API
your-implicit-consumer.properties
your-security.properties

41. 41
KafkaAppender
ELK
KAFKA
Microservice A
Logback
File
Appender
Kafka Appender
Kafka Producer
Logstash
Fordwarder

42. 43
Mutual TLS - your-security.properties
# The location of the key store file.
kafka.tls.keystore.location=/full/path/to/a/keystore.jks
# The store password for the key store file.
kafka.tls.keystore.password=some-good-password
# The file format of the key store file.
kafka.tls.keystore.type=JKS
# The password of the private key in the key store file
kafka.tls.key.password=the-private-keys-password
# The location of the trust store file.
kafka.tls.truststore.location=/path/to/truststore.p12
# The password for the trust store file.
kafka.tls.truststore.password=some-other-good-password
# The file format of the trust store file
kafka.tls.truststore.type=PKCS12
© ING 2016

43. 44
Serialize
Producer
1
Metadata
N
Metadata
1 N
Transformer Serializer
MetadataMetadata
KAFKA1 2 3

44. 45
Filter
Microservice A
Kafka
Producer
Microservice B
Kafka
Consumer
KAFKA
Topic A
Kafka Consumer Kafka Producer
Metadata
Topic B
Filter

45. Transformer
46
Transformer
Microservice A
Kafka
Producer
Microservice B
Kafka
Consumer
KAFKA
Topic A Topic B
1
Kafka Consumer Kafka Producer
Metadata
1 2
1 2
2

46. 47
Forward & Dispatch
Microservice A
Kafka
Producer
Microservice B
Kafka
Consumer
KAFKA
Topic A
Topic C
Kafka Consumer
Metadata
Topic B
Topic N
Kafka Producer
1
1
1 1
Microservice D
Kafka
Consumer
Microservice Z
Kafka
Consumer
Forward & Dispatch

47. 48
Replay
0 1 2 3 i NN-1
beginning i end
Microservice A

48. 49
Performance
Features

49. 50
Performance
Features
FILTERINGDISPATCH

50. 51
Performance
FeaturesFILTERINGDISPATCH

51. Use microservices with Kafka …
… code ahead of the curve
László-Róbert ALBERT
Dan BALESCU Casablanca • 02 November 2016
@danbalescu https://www.linkedin.com/in/albertlaszlorobert