Can and should Apache Kafka replace a database? How long can and should I store data in Kafka? How can I query and process data in Kafka? These are common questions that come up more and more. This session explains the idea behind databases and different features like storage, queries, transactions, and processing to evaluate when Kafka is a good fit, and when it is not. The discussion includes different Kafka-native add-ons like Tiered Storage for long-term, cost-efficient storage, and ksqlDB as an event streaming database. The relation and trade-offs between Kafka and other databases are explored to complement each other instead of thinking about a replacement. This includes different options for pull and push-based bi-directional integration.

1. Building stateful applications
using EDA
- A use case walk-through using Kafka
1
Premjit Mishra
Technology Evangelist
E: premjitmishra@boomi.com

2. 2
Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved.
App Built on Low Code and Kafka
Disclaimer: The views mentioned in this presentation may not reflect my organization’s views or reflect new features in Kafka.
View
Controller Model
Request / Response
OSS SDK

3. 3
Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved.
What was the motivation to use Kafka?
We had a MVC solution to build which needed:
(1) A real time event platform.
(2) A DB to persist some of the work-flows states.
We had a Kafka installation already, so why not try it instead of provisioning for a
DB.

4. 4
Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved.
Extractor Service
GCP NL API
Insights Service
KSQL
GCP NL
automaticTopic outboundTopic
manualTopic reportingTopic
➢ Leads
➢ Opportunity
➢ Account Feed
➢ Case
Application stack with Boomi & Kafka
Connect
Service
Intelligence
Service
Analytics/Data
Service
Synchronize
Service
Tickets

5. 5
Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved.
Dashboard Worklist
Workflow Data Lineage
Intelligent Router App

6. 6
Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved.
What worked well?
➢Agility
The ability to add a new feature quickly was the most important advantage we
found.
➢Realtime Integration and Storage in one platform
Take advantage of Events/Streaming + Storage for realtime integration with
source and sink systems in one platform.
➢Schema Support
We could give a structure to our events using JSON, publish it and use it in our
apps for publishing and consuming from Kafka.

7. 7
Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved.
So, what were the challenges
(DB vs Kafka)?
Traditional Database Kafka
• Has Updates • How do I update in an append only log?
➢Used Offset Commit
➢Topic chaining.

8. 8
Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved.
manualTopic automaticTopic
How we managed Update for
workitems in UI?
Completed Worklist
Pending Worklist
0 1 2 3
0
Commit Offset

9. 9
Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved.
Continue..
Traditional Database Kafka
Has Updates. How do I update in an append only log ?
➢ Used Offset Commit
➢ Topic chaining.
Has SQL features like WHERE, JOIN and aggregate
functions like SUM, AVG etc.
Used ksql TABLES for grouping functions like SUM, GROUP BY.
Ability to use ORDER clause Ordering not possible if # of partitions >1
Indefinite time for storage Keep the operational data for X days and move to data
warehouse for longer duration,reporting and audit requirements.
Theoretically, we can also have indefinite retention period in
Kafka.
Well defined ACID properties and Transaction Management. As it’s an append only log, transactional commit is not possible.

10. 10
Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved.
Key Take Aways
➢ Prototyping helps to unlock new product features or try new ways of doing
things, quickly. Low code helps.
➢ Kafka with DB.

11. 11
Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved.