1. BUILDING KAFKA-POWERED
MESSAGE BUS
Activity Stream and beyond

2. CHALLENGES
➤ Needed a way to aggregate project-related activity information
generated by various services in one place
➤ Needed a messaging backbone to carry messages between
services

3. SOLUTIONS
➤ Activity Stream on the product side
➤ Message Bus on the tech side

4. “A Message Bus is a combination of a
common data model, a common
command set, and a messaging
infrastructure to allow different
systems to communicate through a
shared set of interfaces.
- "Enterprise Integration Patterns"

5. Apache Kafka
is publish-subscribe messaging
rethought as a distributed commit log.
by

6. WHY KAFKA?

7. KAFKA IS FAST
A single Kafka broker can handle hundreds of megabytes of
reads and writes per second from thousands of clients.

8. KAFKA IS SCALABLE
➤ Kafka is designed to allow a single cluster to serve as the
central data backbone for a large organization.
➤ It can be elastically and transparently expanded without
downtime.
➤ Data streams are partitioned and spread over a cluster of
machines to allow data streams larger than the capability of
any single machine and to allow clusters of co-ordinated
consumers.

9. KAFKA IS DURABLE
➤ Messages are persisted on disk and replicated within the
cluster to prevent data loss.
➤ Each broker can handle terabytes of messages without
performance impact.

10. KAFKA IS DISTRIBUTED BY DESIGN
Kafka has a modern cluster-centric design that offers strong
durability and fault-tolerance guarantees.

11. GENERAL SETUP

12. READING / WRITING TO A TWO-PARTITION TOPIC

14. CONSUMER GROUPS

15. MESSAGE BUS GEM
A simple Ruby interface to Apache Kafka.
https://github.com/crealytics/message_bus

16. Camato Shopping
… service
Mesosphere
GewgawConsumer
Bishop
OUR SETUP
Message
Bus
Producer
Message
Bus
Producer
… service
Message
Bus
Producer
Kafka
cluster
Message
Bus
Consumer
TrafficDirector run
DBActivity
Stream
… service
Message
Bus
Consumer
event message
event
message
persistence

17. ACTIVITY STREAM

18. NEXT STEPS
➤ Keep adding messages (and message types) to Activity Stream
➤ Keep improving MessageBus gem
➤ Ensure proper production-ready Kafka setup
➤ Use Message Bus for decoupling service APIs and inter-
service communication
➤ Asynchronous only, needs more research
➤ Input from Martin Mauch: use CrealyticsLogger to push JSON
messages to Kafka (?) — under consideration

19. DEMO

20. DANKE!