Should we use traditional REST APIs to bind services together? Or is it better to use a more loosely-coupled protocol? This talk will dive into how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which.
Building Event-Driven (Micro)Services with Apache Kafka
1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF
HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
Building event-driven Microservices
with Kafka Ecosystem
Guido Schmutz
London, 30.5.2018
@gschmutz guidoschmutz.wordpress.com
2. Guido Schmutz
Working at Trivadis for more than 21 years
Oracle ACE Director for Fusion Middleware and SOA
Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
Head of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 30 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
Building event-driven Microservices with Kafka Ecosystem
3. Agenda
1. Where do we come from?
2. What are Microservices?
3. Why not Event Driven?
4. What about streaming sources?
5. What about integrating legacy applications?
6. What about (historical) data analytics?
7. Why Kafka for Event-Driven Microservices?
8. Summary
Building event-driven Microservices with Kafka Ecosystem
5. Shop Rich UI
Shop Backend Application
Traditional Approach
Search Facade
Customer DAO
Order DAO
Order Facade
Shop UI
Product DAO
UI Logic
DataBusiness
GUI
Customer Fat Client App
Customer BOCustomer UI
DataGUI
Data
Storage
Shared
Database
Building event-driven Microservices with Kafka Ecosystem
sync request/response
6. Shop UI App
Business
Activity Service
SOA Approach
Building event-driven Microservices with Kafka Ecosystem
Contract-first
Web Services
Technical layers
offer their own
interfaces
Reuse on each
level
Lower layer
often wraps
legacy code
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI
Product DAO
UI Logic
GUI
Business Entity
ServiceShop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
7. Shop UI App
Business
Activity Service
Virtualized SOA Approach
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI UI Logic
GUI
Business Entity
Service
Shop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
Service Virtualization Layer
Service Bus
SOAP SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
Building event-driven Microservices with Kafka Ecosystem
9. What are Microservices?
Building event-driven Microservices with Kafka Ecosystem
Tightly Scoped behind clear interfaces
Responsible for managing their own data (not necessarily the infrastructure)
Should be highly decoupled
Independently deployable, self-contained and autonomous
SOA done right ?!
10. Microservice Approach
Building event-driven Microservices with Kafka Ecosystem
Tightly Scoped
Responsible for managing their own data
Highly decoupled
Independently deployable, self-contained
and autonomous
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
11. Microservice Approach
with API Gateway
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway
Building event-driven Microservices with Kafka Ecosystem
12. Synchronous World of Request-Response leads to tight,
point-to-point couplings
Building event-driven Microservices with Kafka Ecosystem
problem in lower end of chain have a
ripple effect on other service
• crash of service
• overloaded service / slow response time
• change of interface
Service 2Service 1
{ }
API
Logic
{ }
API Logic
StateState
Service 3
{ }
API Logic
State
Service 4
{ }
API Logic
State
Service 5
{ }
API Logic
State
Service 7
{ }
API Logic
State
Service 6
{ }
API Logic
State
14. 3 mechanisms through which services interact
Request-Driven (Imperative) Event Driven (Functional)
Service
Logic
State
Event
Consume
OrderEvent{iPad}
Command
”Order IPad”
boolean order(IPad)
Event
Publish
OrderValidatedEvent{iPad}
Query
”Retrieve my Orders
List<Orders> getAllOrders(for)
Building event-driven Microservices with Kafka Ecosystem
15. Event-Driven (Async)
Microservice Approach
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway
Building event-driven Microservices with Kafka Ecosystem
Event
Store
sync request/response
async request/response
async, event pub/sub
16. What about streaming sources?
Building event-driven Microservices with Kafka Ecosystem
17. How to work with
streaming data sources
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
Event
Store
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Event Stream
Building event-driven Microservices with Kafka Ecosystem
18. Hadoop Clusterd
Hadoop Cluster
Stream Processing
Cluster
Streaming Analytics Architecture
BI Tools
SQL
Search / Explore
Online & Mobile
Apps
Search
Service
Event
Stream
Results
Stream Analytics
Reference /
Models
Dashboard
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Microservice Cluster
Microservice State
{ }
API
Event
Stream
Event
Stream
Event
Store
Service
Building event-driven Microservices with Kafka Ecosystem
19. What about integrating legacy
applications?
Building event-driven Microservices with Kafka Ecosystem
20. Data Store
Integrate existing systems through CDC
Customer
Event Store
Integration
Microservice
StateLogic
CDC
CDC Connector
Customer Fat Client App
Customer BOCustomer UI
Stream Processing
Results
Stream Processor
Reference /
Models
Dashboard
Capture changes directly on database
Change Data Capture (CDC) => think like a
global database trigger
Transform existing systems to event
producer
Building event-driven Microservices with Kafka Ecosystem
21. Hadoop Clusterd
Hadoop Cluster
Stream Processing
Cluster
Integrate existing systems through CDC
BI Tools
SQL
Search / Explore
Online & Mobile
Apps
Search
Service
Event
Stream
Results
Stream Processor
Reference /
Models
Dashboard
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Microservice Cluster
Microservice State
{ }
API
Event
Stream
Event
Stream
Event
Store
Service
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Change Data
Capture
Building event-driven Microservices with Kafka Ecosystem
22. What about (historical) data
analytics?
Building event-driven Microservices with Kafka Ecosystem
23. Streaming & (Big) Data Analytics Architecture
Event
Stream
Hadoop Clusterd
Hadoop Cluster
Big Data Cluster
Data
Flow
Parallel
Processing
Storage
Storage
RawRefined
Results
Microservice Cluster
Microservice State
{ }
API
Stream Processing Cluster
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
SQL
Export
SearchEvent
Store
Service
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Change Data
Capture
File Import / SQL Import
Building event-driven Microservices with Kafka Ecosystem
24. Why Kafka for Event-Driven
Microservices?
Building event-driven Microservices with Kafka Ecosystem
25. Apache Kafka – A Streaming Platform
Building event-driven Microservices with Kafka Ecosystem
High-Level Architecture
Distributed Log at the Core
Scale-Out Architecture
Logs do not (necessarily) forget
28. Legacy Microservice
Change Data Capture (CDC)
Building event-driven Microservices with Kafka Ecosystem
RDBMS cdc-source trucking_
driver
Customer Topic
elasticsearc
h-sink NoSQL
29. Demo
Building event-driven Microservices with Kafka Ecosystem
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
REST
REST
Kafka
Customer
30. Hold Data for Long-Term – Data Retention
Producer 1
Broker 1
Broker 2
Broker 3
1. Never
2. Time based (TTL)
log.retention.{ms | minutes | hours}
3. Size based
log.retention.bytes
4. Log compaction based
(entries with same key are removed):
kafka-topics.sh --zookeeper zk:2181
--create --topic customers
--replication-factor 1
--partitions 1
--config cleanup.policy=compact
Building event-driven Microservices with Kafka Ecosystem
31. Topic Viewed as Event Stream or State Stream (Change
Log)
Event Stream State Stream (Change Log Stream)
2015-10-02 11,Peter,Muster,3010,Berne
2016-10-04 12,Paul,Steffen,8001,Zurich
2016-12-02 21,Lisa,Meier,3043,Ittigen
2017-05-03 11,Peter,Muster,3015,Berne
2017-05-03 21,Lisa,Steffen,8001,Zurich
2017-07-03 11,Peter,Muster,3052,Zollikofen
Building event-driven Microservices with Kafka Ecosystem
2015-10-02 11,Peter,Muster,3010,Berne
2016-10-04 12,Paul,Steffen,8001,Zurich
2016-12-02 21,Lisa,Meier,3043,Ittigen
2017-05-03 11,Peter,Muster,3015,Berne
2017-05-03 21,Lisa,Steffen,8001,Zurich
2017-07-03 11,Peter,Muster,3052,Zollikofen
33. Keep Topics both in Original and Compacted Form
Building event-driven Microservices with Kafka Ecosystem
Kafka Broker
A B C B A E K A E A C K
B E A C K
Producer
Kafka Broker
A B C B A E K A E A C K
B E A C K
Producer
Consume and Produce
OR
TX
34. Enrich Stream with Static Data with Kafka Streams
Building event-driven Microservices with Kafka Ecosystem
Order Topic Consume and Produce
Customer
Table
Customer Topic
Custome
r Handler
Join
Customer Local State
Order & Customer Topic
36. Hadoop Clusterd
Hadoop Cluster
Big Data
Summary
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
File Import / SQL Import
Event
Hub
Data
Flow
Data
Flow
Change
Data
Capture
Parallel
Processing
Storage
Storage
RawRefined
Results
SQL
Export
Microservice State
{ }
API
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
Search
Service
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Stream Processing
Microservices
37. Summary
Building event-driven Microservices with Kafka Ecosystem
• service autonomy is key in a Microservices Architecture!
• not all communication need to be synchronous => separate into
• commands
• events
• queries
• Kafka is well suited as an event broker / event store
• brings many more interesting features beyond just “message passing”
38. References
Building event-driven Microservices with Kafka Ecosystem
Microservices Blog Series, Ben Stopford, Confluent:
• https://www.confluent.io/blog/tag/microservices
Apache Kafka for Microservices: A Confluent Online Talk Series:
• https://www.confluent.io/landing-page/microservices-online-talk-series/
Turning the database inside-out with Apache Samza, Martin Kleppmann, Con
• https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/
Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent:
• https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/
Immutability Changes Everything, Pat Helland, Salesforce:
• http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood:
• https://www.youtube.com/watch?v=B1-gS0oEtYc
39. Technology on its own won't help you.
You need to know how to use it properly.
Building event-driven Microservices with Kafka Ecosystem