Stream processing allows analyzing streaming data in real-time using tools like KSQL. KSQL builds on Kafka Streams to provide a SQL-like interface for transforming, filtering, and creating derived streams from immutable event data ingested into Kafka. Jupiter, a digital banking platform, uses KSQL to reconcile transactions from different interfaces in real-time and check rewards eligibility by analyzing consolidated transaction streams. While powerful, KSQL deployments require monitoring, richer SQL support, and integration with big data platforms for long-term storage.

1. Presentation
Header here
Stream Processing via Kafka and KSQL
@namit_mahuvakar #kafkaSummit2021

2. 02. What is Stream Processing?
table of
contents
01. About Jupiter
03. Stream Processing with KSQLDB
04. KSQLDB in Jupiter
06. KSQL in the current Data Ecosystem

3. About Us
Jupiter is a digital banking experience that
provides:
● Digital banking experience for the millennials
in India
● Re-imagining and re-building different banking
functions, with technology-first approach
● A financial platform that constantly calculates
and improves the financial wellness of users
● Smarter savings, Transparency, Personalised
rewards

4. Banking Experience

5. What is Stream Processing?
Logs
DB events
Customer Experience
IoT
Transactions and
Purchases

6. What is Stream Processing?

7. Stream Processing with KSQL
● Immutable events being ingested
● KSQL built on top of Kafka Streams
● Continuous Query Language in SQL
● Ease of Transformation, filtering
● Creating derived streams

8. Stream Processing with KSQL
create stream events_a as
select * from events
where type = "A"

9. Transaction Reconciliation
Jupiter as a banking platform supports multiple
types of transaction interfaces -
● UPI: (Unified Payments Interface)
● Bank Transfers: (NEFT, IMPS, RTGS, IFT)
● Card Transfers: (Debit Card Transactions)
Transactions on certain interfaces can be
initiated on the application.

10. Transaction Reconciliation
Problem Statement : Need to aggregate all
events to enrich the individual transaction
record.
Sources -
● Interface Callbacks
● Bank Accounting Systems
Challenges - Time delay between
events/Synchronisation ?

13. Demo

15. Rewards Eligibility
Problem Statement: Utilise incoming streams of
data for,
● Check rewards eligibility
● Give 10% Jewels for all UPI transactions
● Provide specific rewards to those eligible
● Utilise already consolidated derived stream

17. Demo

18. What more can you do ?

19. Stream Processing with KSQL

20. Stream Processing with KSQL

23. KSQLDB Operations and Challenges
● Deployment - Deploy via Docker/K8s, images
available at docker hub
● Tear down stack
● Monitoring - Enable explicitly for JMX metrics
to be available
● Richer SQL grammar required
● Point in time select queries
● Checkpointing every 100ms
● Granular insights on consumer and stream
performance

24. KSQL the one for the future ?
Data consumption in real time allows us to act
faster on opportunities and avoiding issues as
and when they occur,
● KSQL allows anyone with SQL knowledge to
process any data
● No need to write any code
● Need for real time data in an abstract
environment is solved easily i.e. L0 metrics for
platform alerting
● Need for integration with Big Data Platforms,
spill to disk or directly to Data Lake

25. Thank you,
Any Questions ?

26. Use Case :
Transaction Reconciliation

27. Use Case :
Rewards Eligibility

28. Stream Processing with KSQL