Event Driven Architecture and Apache Kafka were discussed. Key points:
- Event driven systems allow for asynchronous and decoupled communication between services using message queues.
- Apache Kafka is a distributed streaming platform that allows for publishing and subscribing to streams of records across a cluster of servers. It provides reliability through replication and allows for horizontal scaling.
- Kafka provides advantages over traditional queues like decoupling, scalability, and fault tolerance. It also allows for publishing of data and consumption of data independently, unlike traditional APIs.
2. Agenda
● Introduction to Event Driven Systems
● Introduction to different problems that arise when using Event Driven Systems
● Different patterns
● Apache Kafka
● Org-wide implementations
3. Lets Build Scamazon
Disclaimer: The session will have stupid poor jokes scattered all over. Forgive the incompetent joker in me.
4. Developer report
My code provisions the order
quickly. While investigating, I
observed that sending the order
successful mail takes a lot of
time.
Customer report
Placing an order on
Scamazon is such a pain.
It takes too long.
Latency: Amount of time it takes to complete one request
Throughput: Amount of requests that can be completed in a unit of time
6. Scamazon Customer report
I placed an order on the last
samsong smart watch before
my friend yesterday. But he got
the watch and I didnt.
Annoying Business manager
What? How can that happen?
Devs, fix it.
7. Transaction Ordering
1. Have a Distributed Locking
mechanism
2. Ensure Message queues
inherently supports ordering
8. Customer report
On your Great Scamazon
sale, I purchased a shoe. I
never got it.
Apparently Microservices
are in trend now!
Business Manager
Each customer we lose, is
one gained for FlopKart! Jeff
Bozo is going to kill us. Devs,
Do Something !!
Developer
Delivery Service was down
because some SysAd
upgraded something.
Blame them! My code is gold.
11. Message queues
1. Message queueing allows applications to communicate
asynchronously by sending messages to each other.
2. Messages may or may not be sent in FIFO
12. Message queues v/s APIs
APIs are 2-way communication.
Message Queues are 1-way communication.
13. Some theory
● Messages: Messages can be commands or events. Each message can have a type
and payload data to be sent across.
● Channels/Queues: Channels/Queues are delivery points the messages are sent to.
● Event Dispatcher/Producers: The duty of the dispatcher is to register channels and
produce events/messages.
● Event Handlers/Consumers: Event handlers act as destination points for receiving
events as channels do.
● Dynamic Routers: The harmony of messaging systems occurs through its routers.
Routers are responsible for selecting the proper path for a given message.
14. Review Message Queue Advantages
1. Decoupled Architecture: The sender’s job is to send messages to the queue
and the receiver’s job is to receive messages from the queue. This separation
allows for components to operate independently, while still communicating
with each other.
1. Fault tolerant: If the consumer server is down, the messages would still be
stored in queue until the server goes back up and retrieves it
2. Scalability: We can push all the messages in the queue and configure the rate
at which the consumer server pulls from it. This way, the receiver won’t get
overloaded with network requests and can instead maintain a steady stream
of data to process.
3. Transaction Ordering: Have patterns to Ensure messages are delivered in
order of they getting published if the use-case needs it
18. Event Sourcing
Events may or may not be stored in
an EventStore(ex: broker).
4
Designed for Fast & low latency
applications(ex: stock trading)
1
Capture all changes to an application
state as a sequence of events.
2
Events define the state of the
system.
3
Like git, it doesn't store the latest
version of code, but maintains the
changes from the base version.
5
If server goes down, we can replay
the events and get the state back.
6
19. Event sourcing Advantages/ Disadvantages
Advantages
1. Entire applications can be in
memory
2. Parallel model
Disadvantages
1. Event schemas
2. Non event sourcing service
interaction
20.
21. Apache Kafka
● Developed by LinkedIn.
● Building real-time streaming data pipelines that reliably
get data between systems or applications
● Building real-time streaming applications that transform
or react to the streams of data
1. Channels a.k.a Topics
2. Brokers a.k.a Routers
Distributed Streaming Platform
22. Records in Kafka
Each Record(Message) in kafka has:
1. Key
2. Value
3. Timestamp
Key
Value
Timestamp
null
{“orderid”:1,
“cid”: 5}
1322468906767
8461
Abhishek
1592468905404
resellerhosting
{“cid”: 5,
“addons”: []}
1492468905404
Examples:
23. Topic
1. Partition: Ordered, immutable
sequence of records that is
continually appended to
2. Offset: Uniquely identifies each
record within the partition
24. How is kafka resilient ?
1. Cluster of kafka servers
2. Partitions are replicated in all
3. Each Server function as leader for a fair share
of partitions
4. Followers passively replicate leader.
5. Each kafka server will have leaders and
followers.
P1
Broker 1
P1 P2 P3
P2 P3
Broker 2
25. Can you Explain?
Scenario 1:
Brokers: 3
Partitions: 3
Replication Factor: 2
Scenario 2:
Brokers: 3
Partitions: 2
Replication Factor: 3
Broker 1
P
1
P
3
Broker 2
P
2
P
1
Broker 3
P
3
P
2
Broker 1
P
1
P
2
Broker 2
P
2
P
1
P
1
Broker 3
P
2
26. APIs provided by Kafka
1. Producer: publish messages to one or more topics
2. Consumer: subscribe to a topic and process these messages
3. Connect API: Bind existing systems like databases to kafka topics
4. Streams API: Consume a stream of data and produce one or more output streams.
5. Admin API: manage topics, broker and other kafka objects
28. Kafka Producers
● Producers publish records to the kafka topics with help of kafka provided kafka APIs
● Any number of producer can public records to the same topic
● Producers can decide the partition of the topic into which record to publish
○ For normal use case and better load balancing they follow round-robin allocation
● Batch processing on producer side. Why and at What cost?
● How records are routed from producer to the leader of target partition?
29. Why Zookeeper?
● Zookeeper can provide primitives to support:
○ Distributed Configuration Service
○ Synchronization
○ Cluster Management
○ Service Discovery
● Highly optimized for reads than writes.
● How kafka utilizes Zookeeper?
○ Controller Election
○ Configuration of Topics
○ Membership of the cluster
Coordination Service for Distributed Applications
31. Why is Kafka not a simple queue ?
1. durably persists all published
records—whether or not they have
been consumed—using a
configurable retention period
2. This offset is controlled by the
consumer
3. Consumer can consume however it
wants.
4. Consumer can reset to an older
offset to reprocess data or skip
ahead.
5. Kafka provides feature of both
Queuing and Pub-Sub model with its
partitioned topic based design.
32. Ordering of Records in Kafka
● Total ordering over records within partition
● How to achieve total ordering over all data?
● Why kafka decided to provide ordering over
only partition?
Kafka Provides better ordering than Queues!!?
34. Example of Enterprise Messaging in Endurance
BLL implemented event driven architecture using Kafka.
1
RP uses ActiveMQ’s Queues and Topics for messaging,
notifications etc.
2
CA uses it for CQRS via ActiveMQ.
4
OrderBox uses Kafka for Streaming changes from databases.
3
38. Why Kafka for Data Streaming(why not ActiveMQ) ?
● Plethora of connectors to connect to multiple data sources.
● Inherent architecture promotes its use as a Log store.
● Kafka Streams provides an amazing set of API’s to provide aggregation features.
● This format of streaming is abbreviated as Change Data Capture
○ More details can be found here
39. RP use cases
1. Message from Core Web layer to
Executor Layer(Queue)
2. Cache Eviction across containers of
same service(Topic)
3. Cache eviction across different
services(Topic)
4. Can use Event Carried State
Transfer for cache updation(Topic)
