We now live in a world with data at its heart. The amount of data being produced every day is growing exponentially and a large amount of this data is in the form of events. Whether it be updates from sensors, clicks on a website or even tweets, applications are bombarded with a never-ending stream of new events. So, how can we architect our applications to be more reactive and resilient to these fluctuating loads and better manage our thirst for data? In this session explore how Kafka and Reactive application architecture can be combined in applications to better handle our modern data needs.

1. Reacting to an Event-
Driven World
Grace Jansen
IBM Developer Advocate
@gracejansen27
© 2021 IBM Corporation
Kate Stanley
IBM Software Engineer
@katestanley91

2. Let’s get some coffee…
© 2020 IBM Corporation

3. Barista Example:
© 2020 IBM Corporation
Coffee Shop
Barista
Coffee Lovers
HTTP
HTTP
https://github.com/cescoffier/quarkus-coffeeshop-demo

4. Event Driven Architecture
© 2020 IBM Corporation
S1
S3
S2
S4
Event-driven messaging backbone
Microservice
publishing
events
Microservice
consuming
events
Event-driven messaging backbone
http://ibm.biz/AdvantagesOfEDA

5. Barista Example:
© 2020 IBM Corporation
Coffee Shop
Coffee Shop
Barista
Barista
Board
Coffee Lovers
Coffee Lovers
HTTP
Orders,
Queue
HTTP
HTTP
https://github.com/cescoffier/quarkus-coffeeshop-demo
Event
Backbone

6. © 2020 IBM Corporation
Apache Kafka is an open source, distributed
streaming platform
Stream history Immutable data Highly available
Scalable
Scalable
consumption

7. © 2020 IBM Corporation
Q: Is your coffee shop non-blocking and
highly responsive?

8. © 2020 IBM Corporation
Q: Is your microservice system non-
blocking and highly responsive?

9. © 2020 IBM Corporation
Q: Is your microservice system non-
blocking and highly responsive?
A:Yes I’m using Kafka!

10. © 2020 IBM Corporation
Q: Is your microservice system non-
blocking and highly responsive?
A:Yes I’m using Kafka!

11. Reactive systems
© 2020 IBM Corporation

12. Reactive Manifesto
© 2020 IBM Corporation

13. Message-Driven
Reactive Manifesto
© 2020 IBM Corporation

14. Events
“A signal emitted by a component upon
reaching a given state.”
© 2020 IBM Corporation
A message can contain an encoded
event in its payload.
Messages
“An item of data sent to a specific location.”

15. © 2020 IBM Corporation
Apache Kafka is an open source, distributed
streaming platform
Publish and subscribe to streams of
records
Store records in durable way
Process streams of records as they
occur

16. Barista Example:
© 2020 IBM Corporation
Coffee Shop
Barista
Board
Coffee Lovers
Orders,
Queue
HTTP
https://github.com/cescoffier/quarkus-coffeeshop-demo
Event
Backbone

17. Barista Example:
© 2020 IBM Corporation
Coffee Shop
Barista
Board
Coffee Lovers
Orders,
Queue
HTTP
https://github.com/cescoffier/quarkus-coffeeshop-demo
Event
Backbone

18. Barista Example:
© 2020 IBM Corporation
Coffee Shop
Barista
Board
Coffee Lovers
Orders,
Queue
HTTP
https://github.com/cescoffier/quarkus-coffeeshop-demo
Event
Backbone

19. Elastic
Message-Driven
Reactive Manifesto
© 2020 IBM Corporation

20. Barista Example:
© 2020 IBM Corporation
Coffee Shop
Barista
Board
Coffee Lovers
Orders,
Queue
HTTP
https://github.com/cescoffier/quarkus-coffeeshop-demo
Event
Backbone

21. Barista Example:
© 2020 IBM Corporation
Coffee Shop
Barista
Board
Coffee Lovers
Orders,
Queue
HTTP
https://github.com/cescoffier/quarkus-coffeeshop-demo
Event
Backbone

22. Barista Example:
© 2020 IBM Corporation
Coffee Shop
Barista
Board
Coffee Lovers
Orders,
Queue
HTTP
https://github.com/cescoffier/quarkus-coffeeshop-demo
Event
Backbone

23. Resilient
Elastic
Message-Driven
Reactive Manifesto
© 2020 IBM Corporation

24. Resilient
Elastic
Responsive
Message-Driven
Reactive Manifesto
© 2020 IBM Corporation

25. Resilient
Elastic
Responsive
Message-Driven
Reactive Manifesto
© 2020 IBM Corporation

26. Building reactive
systems
© 2020 IBM Corporation

27. How do we make a highly responsive app?
Microservice 1
© 2020 IBM Corporation
Microservice 2
Microservice 3

28. How do we make a highly responsive app?
Microservice 1
© 2020 IBM Corporation
Microservice 2
Microservice 3
Reactive
Architecture
Patterns

29. Reactive Architecture design patterns
CQRS Circuit breaker Saga Back pressure
© 2020 IBM Corporation
APP
WRITE API
APP
READ API
APP
APP
APP
APP
APP
APP
APP
APP

30. How do we make a highly responsive app?
Microservice 1
© 2020 IBM Corporation
Microservice 2
Microservice 3
Futures
Reactive Streams
Libraries
Futures
Reactive Streams
Libraries
Futures
Reactive Streams
Libraries
Reactive
Programming

31. Reactive Programming
© 2020 IBM Corporation
A subset of asynchronous programming and a paradigm where the
availability of new information drives the logic forward rather than
having control flow driven by a thread-of-execution.

32. Reactive Programming Patterns
© 2020 IBM Corporation
Futures: a promise to hold the result of some operation once that operation completes
Reactive programming libraries: for composing asynchronous and event-based
programs. (e.g. RxJava, SmallRye Mutiny)
Reactive Streams: a programming concept for handling asynchronous data streams
in a non-blocking manner while providing backpressure to stream publishers

33. Utilising Kafka in
reactive systems
© 2020 IBM Corporation

34. Resilient
Elastic
Responsive
Message-Driven
Message-Driven with Kafka
© 2020 IBM Corporation

35. Resilient
Elastic
Responsive
Message-Driven
Resiliency in Kafka
© 2020 IBM Corporation

36. Message Retention and Data
Persistence
© 2020 IBM Corporation
Stream history Immutable Data

37. Resilient Kafka Clusters
© 2020 IBM Corporation

38. © 2020 IBM Corporation
Resilient Kafka Clusters

39. Resilient Kafka Clusters
© 2021 IBM Corporation

40. Topic A
Partition 1
Broker 1
Topic A
Partition 1
Offline Leader
Resilient Kafka Clusters
© 2021 IBM Corporation

41. Resilient Producers
Delivery guarantees:
At most once
At least once
Configuration:
Acks
Retries
© 2021 IBM Corporation

42. Resilient Consumers
0 1 2 3 4 5
Offset
© 2021 IBM Corporation

43. Baristas with auto commit
Barista
TOPIC
Coffee Cappuccino Latte

44. Baristas with auto commit
Barista
TOPIC
Coffee Cappuccino Latte
Coffee

45. Baristas with auto commit
Barista
TOPIC
Coffee Cappuccino Latte
Coffee

46. Baristas with auto commit
Barista
TOPIC
Coffee Cappuccino Latte

47. Baristas with auto commit
Barista
TOPIC
Coffee Cappuccino Latte
Cappuccino

48. Baristas with auto commit
Barista
TOPIC
Coffee Cappuccino Latte
Cappuccino Latte

49. Baristas with auto commit
Barista
TOPIC
Coffee Cappuccino Latte
Cappuccino Latte
Baristas with manual commit
Coffee Cappuccino Latte
TOPIC
Barista

50. Baristas with auto commit
Barista
TOPIC
Coffee Cappuccino Latte
Cappuccino Latte
Baristas with manual commit
Coffee Cappuccino Latte
TOPIC
Barista
Coffee

51. Baristas with auto commit
Barista
TOPIC
Coffee Cappuccino Latte
Cappuccino Latte
Baristas with manual commit
Coffee Cappuccino Latte
TOPIC
Barista
Coffee

52. Baristas with auto commit
Barista
TOPIC
Coffee Cappuccino Latte
Cappuccino Latte
Baristas with manual commit
Coffee Cappuccino Latte
TOPIC
Barista
Coffee Latte
Cappuccino

53. Schemas and Schema Registries
© 2020 IBM Corporation

54. Resilient
Elastic
Responsive
Message-Driven
Scalability in Kafka
© 2021 IBM Corporation

55. Scalability in Kafka
© 2021 IBM Corporation

56. Elasticity in Consumers
© 2021 IBM Corporation

57. Consumer Groups
Consumer groupA
Topic
0 1 2 3
0 1 2 3 4 5
Partition 0
Partition 1
Partition 2
0 1 2 3 4 5 6 7
Consumer
Consumer
Consumer
Consumer group B
Consumer
Consumer
© 2021 IBM Corporation

58. Consumer Groups
Consumer groupA
p0, offset 7
p1, offset 3
p2, offset 5
Topic
0 1 2 3
0 1 2 3 4 5
Partition 0
Partition 1
Partition 2
0 1 2 3 4 5 6 7
Consumer
Consumer
Consumer
Consumer group B
Consumer
Consumer
© 2021 IBM Corporation

59. Consumer Groups
Consumer groupA
p0, offset 7
p1, offset 3
p2, offset 5
Topic
0 1 2 3
0 1 2 3 4 5
Partition 0
Partition 1
Partition 2
0 1 2 3 4 5 6 7
Consumer
Consumer
Consumer
Consumer group B
p0, offset 7
p1, offset 3
p2, offset 5
Consumer
Consumer
© 2021 IBM Corporation

60. Consumer Groups
Consumer groupA
p0, offset 7
p1, offset 3
p2, offset 5
Topic
0 1 2 3
0 1 2 3 4 5
Partition 0
Partition 1
Partition 2
0 1 2 3 4 5 6 7
Consumer
Consumer
Consumer
Consumer
Consumer group B
p0, offset 7
p1, offset 3
p2, offset 5
Consumer
Consumer
© 2021 IBM Corporation

61. Writing reactive
Kafka applications

62. © 2021 IBM Corporation

63. Reactive Frameworks for
Kafka
Alpakka Kafka Connector
MicroProfile Reactive Messaging
Vert.x Kafka Client
© 2021 IBM Corporation

64. Alpakka Kafka Connector API
© 2021 IBM Corporation

65. Eclipse MicroProfile
An open-source community specification for Enterprise
Java microservices
A community of:
individuals
organizations
vendors
© 2021 IBM Corporation

66. MicroProfile 3.3 Stack
Reactive Streams
Operators 1.1
GraphQL
1.0
Standalone Projects
JSON-B 1.0
JSON-P 1.1
CDI 2.0
Config 1.4
Fault
Tolerance 2.1
JWT
Propagation
1.1
Health
Check 2.2
Metrics 2.3
OpenTracing
1.3
Open API 1.1
JAX-RS 2.1 Rest Client 1.4
Reactive
Messaging 1.0
Context
Propagation
1.0
Reactive
Streams
Not a MicroProfile spec
© 2021 IBM Corporation

67. MicroProfile Reactive Messaging
Application’s beans contain methods annotated with @Incoming and @Outgoing annotations
The annotated methods are connected by named channels
A channel is a name indicating which source or destination of messages is used. Channels are opaque Strings.
@Incoming and @Outgoing annotations are matched up by channel names
© 2021 IBM Corporation

68. What is Eclipse Vert.x?
Polyglot Tool-kit
Based on Reactor pattern
Runs on the JVM
Non-blocking
Event-driven
Includes distributed event-bus
Code is single-threaded
© 2021 IBM Corporation

69. Vert.x Demo App
© 2020 IBM Corporation

70. Demo app
© 2021 IBM Corporation
https://github.com/ibm-messaging/kafka-java-vertx-starter
Vert.x app
produce
consume

71. Demo app
© 2021 IBM Corporation
Vert.x app
websocket
produce
consume
https://github.com/ibm-messaging/kafka-java-vertx-starter

72. © 2021 IBM Corporation
https://github.com/ibm-messaging/kafka-java-vertx-starter

73. © 2021 IBM Corporation

74. Producing Records
© 2021 IBM Corporation
Vert.x app
websocket produce
Start
Stop

75. Producing Records
© 2021 IBM Corporation

76. Consuming and Processing Records
© 2021 IBM Corporation
Apache Kafka Java Client:

77. © 2021 IBM Corporation
Consuming and Processing Records
Apache Kafka Java Client:

78. © 2021 IBM Corporation
Consuming and Processing Records
Apache Kafka Java Client:

79. © 2021 IBM Corporation
Consuming and Processing Records
Vert.x Kafka Client

80. © 2021 IBM Corporation
Consuming and Processing Records
Vert.x Kafka Client

81. © 2021 IBM Corporation
Flow Control
Vert.x Kafka Client

82. © 2021 IBM Corporation ibm.biz/ExperiencesWritingAReactiveKafkaApp

83. Summary
Non-reactive + Kafka != reactive
Consider Kafka configuration for the best reactive system
The open-source reactive community is on hand to help!
Reactive toolkits and frameworks can provide additional benefits
© 2021 IBM Corporation

84. Reactive Systems Explained
© 2021 IBM Corporation
ibm.biz/ReactiveReport

85. MicroProfile Reactive labs
ibm.biz/reactive-java-labs
© 2021 IBM Corporation

86. Thank you
Grace Jansen | @gracejansen27
Kate Stanley | @katestanley91
© 2021 IBM Corporation
Reactive resources:
https://ibm.biz/IntroToReactive
https://ibm.biz/GettingStartedWithReactive
Getting started with Kafka:
https://kafka.apache.org/quickstart
https://strimzi.io
https://ibm.github.io/event-streams/schemas/overview/
Reactive Kafka libraries
https://vertx.io/docs/vertx-kafka-client/java/
https://github.com/eclipse/microprofile-reactive-
messaging
https://projectreactor.io/docs/kafka
https://github.com/akka/alpakka-kafka
Open Liberty:
https://openliberty.io