A brief example that demonstrates asic usage of Kafka and Spring Integration when you build a microservice.

1. Introduction to Kafka
with Spring Integration

2. Introduction to Kafka with Spring
Integration
• Kafka (Mihail Yordanov)
• Spring integration (Borislav Markov)
• Students Example (Mihail & Borislav)
• Conclusion

3. Motivation
• Real time data being continuously generated
• Producers and consumers relationships
• Streaming systems
• Messaging systems

4. Apache Kafka
• Developed by LinkedIn
• “We built Apache Kafka at LinkedIn with a specific
purpose in mind: to serve as a central repository of
data streams.” - Jay Kreps
• Kafka improvements
– Transporting data between systems
– Richer analytical processing
• “A publish-subscribe messaging system rethought as a
distributed commit log”

5. Kafka Vocabulary
• Brokers
• Producers
• Consumers
• “A publish-subscribe messaging system
rethought as a distributed commit log”

6. Kafka Vocabulary
• Topics
• Partitions
• “A publish-subscribe messaging system rethought
as a distributed commit log”

7. Kafka Speed

8. Kafka Speed

9. Kafka Speed

10. Consumer groups
• Definition - “A publish-subscribe messaging
system rethought as a distributed commit log”
• Publish - Subscribe
• Queueing

11. Spring Integration
• Spring Integration enables lightweight
messaging
• Supports integration with external systems via
declarative adapters
• Primary goal is to provide a simple model for
building enterprise integration solutions
• Separation of concerns
• Maintainable, testable code

12. Features
• Implementation of most of the Enterprise Integration Patterns
• Endpoint
• Channel (Point-to-point and Publish/Subscribe)
• Aggregator
• Filter
• Transformer
• Control Bus
• …
• Integration with External Systems
• ReST/HTTP
• ...

13. The theory...
The patterns can be found at
http://www.enterpriseintegrationpatterns.com/patterns/messag
ing/index.html

14. Message
• Messages are objects that carry information between two applications
• They are constructed by the producer and they are consumed/deconstructed at
the consumer/subscriber
• Message consists of:
– payload
– header
public interface Message<T> {
T getPayload();
MessageHeaders getHeaders();
}

15. Message endpoints
Common endpoints:
• Service Activator - invokes a method on a
bean
• Message Bridge - couples different
messaging modes/adapters. Example: (P2P
with Publish/Subscribe)
• Message Enricher - enrich the incoming
message with additional information.
– Header Enricher - add header
attributes
– Payload Enricher - enrich payload

16. Message endpoints (continued)
• Gateway - Used when we don’t have knowledge for the
messaging system.
– Synchronous Gateway - void or returns T
– Asynchronous Gateway - returns Future<T>
• Delayer - introduce delay between sender and receiver
• Consumers
– Polling Consumers - poll messages in a timely fashion
– Polling Using Triggers - poll messages with PeriodicTrigger
and with CronTrigger
– Event-Driven Consumers - they wait for someone to
deliver the message (framework’s responsibility)

17. Message endpoints (continued)
• Channel Adapter - endpoint that connects a Message Channel
to some other system or transport.
–inbound
–outbound

18. Message Channels
• interface MessageChannel
– boolean send(Message<?> message);
– boolean send(Message<?> message, long timeout);
• Point-to-Point Mode -> PollableChannel
– Message<?> receive();
– Message<?> receive(long timeout);
• Publish/Subscribe Mode -> SubscribableChannel
– boolean subscribe(MessageHandler handler);
– boolean unsubscribe(MessageHandler handler);

19. Message Channels(continued)
•Queue Channel - has capacity
•Priority Channel - queue channel with
prioritization
•PublishSubscribe Channel
•DirectChannel - mixed type P2P and
Pub/Sub

20. Message Channels(continued)

21. Flow components
• Filters
– Custom filters are tied to the framework, can
be bean method returning boolean
– Framework MessageSelector - use of method
boolean accept(Message m)
– Using annotation - @Filter
• Routers
– PayloadTypeRouter
– HeaderValueRouter
– Custom Routers - any bean method can be
router, the result will be the channel name.
– Recipient List Router - the message goes to all
statically defined channels

22. Flow components (continued...)
• Splitters - splits a message into pieces
– Custom splitters - any bean method;
– Framework AbstractMessageSplitter - use of method Object splitMessage(Message
m);
– Using annotation - @Splitter ;
• Aggregator
– assemble multiple messages to create a single parent message
– Complex task - all messages of a set have to arrive before aggregators can start work
– CorrelationStrategy - defines the key for grouping of the messages, the default
grouping is based on CORRELATION_ID
– ReleaseStrategy - dictates at which point the group of messages should be sent or
released for aggregation. Default is SequenceSizeReleazeStrategy
– Message Store - aggregators hold messages until all of them arrived. Options are:
• in-memory
• external database
• Resequencer - fix order of the messages(work on SEQUENCE_NUMBER)

23. Transformers
• Built-in transformers
– String Transformers - invokes toString() method of the payload
– Map Transformers - transformes POJO to a name-value pair of
Map
– Serializing and Deserializing Transformers - converts payload
to byte array
– JSON Transformers - object-to-json converts POJO to readable
JSON format; json-to-object transformer needs additional
property “type”
– XML Transformers - requires Spring OXM (Object-to-XML);
org.springframework.oxm.Marshaller/Unmarshaller

24. Transformers(continued...)
• Built-in transformers
– XPath Transformers - decodes
XML using XPath expressions, ex.:
xpath-
expression=”/mytag/@prop”
• Custom Transformers - can be any
spring bean method
• Using @Transformer

25. Students Example
• Example application will demo usage
of Kafka and Spring Integration
• App is built with maven
• Ideal candidate for Microservice
• Idea: takes students from outside and
calculates their average score

26. Maven dependencies

27. Project Structure
• Package “api” – POJO models
• Package “config” – beans with
factory methods
• Package “service” – the
business logic
• Folder “resources” –
application.properties

28. Spring Integration DSL

29. Students Example

30. Service activator

32. Running the app and giving some
inputs

33. Check the logs

34. Conclusion
Why Kafka and Spring ?
• Easy for microservices
• Clean and testable code
• Widely adopted technologies
• Easy to be dockerized