"Financial services organizations need robust event streaming platforms. Optimizing client application parameters when connecting to the streaming platform can be challenging. At Fidelity Investments, we built a streaming platform that hosts business application events for event driven architectures and integration with other business applications through events and streams. We simplified this process through Kafka client-side governance automation to help application development teams configure parameters compliant with security best practices and to optimize performance and throughput with an OPA rules engine plus AWS ML model prediction. In this session, we will share the following aspects with a demo of linting APIs: · Challenges with Kafka client-side governance · OPA setup · AWS ML models for prediction · Demo of Kafka client-side linting You will learn ideas for client governance and linting of Kafka client application as part of this talk. Kafka client governance is essential for the smooth operation of a financial services organization and for maintaining the trust of its customers."

1. Using Machine Learning to Govern
Kafka Clients
UmaMahesh Sistu & Shu Wang
Event Streaming Platform Team, Fidelity Investments

2. PART 1 - Introduction
Streaming platform
Application structure
Initial ideas
Application governance
PART 3 – Machine Learning
Machine learning
Continuous learning
Optimization
PART 2 - Implementation
Implementation
Challenges
Kafka linting tool
Rules engine
Demo
Objectives
Using Machine Learning to Govern Kafka Clients

3. 3
Part 1
Introduction
Streaming Platform Ecosystem of Services

4. Fidelity Management &
Research Company
Founded
Fidelity Purchases
First Computer
Fidelity Becomes an
Internet Pioneer with
First Mutual Fund to
Create a Home Page
Today
6,500+ Applications on Public Cloud
Fidelity’s First
Product Application
Deployed to the
Cloud
1965
1946
1995
2016
2019
Fidelity Launches Multi-Cloud Hybrid Strategy
Fidelity’s Cloud Journey

5. Event Streaming Platform
4
Years in
public cloud
6B+
Events per day
72+
16k+
Producer and consumer
Applications
Self-service APIs
300+
Observability Metrics

6. Event Streaming Platform
Core and Common Capabilities
Schema Registry
Connectors
Query and Analyze Streaming
Data
Management Plane
Events and Streams
Control
Plane
Data Plane
Tech
Stack
Telemetry
Database
CRM
Other
Data
Warehouse
Down Stream
Apps
DATA
INTEGRATION
App Events Log Events Mobile IoT Events
Business
Events
Analytics & ML
Inventory
Management
Other
Fraud
Detection
Transactional
Apps
REAL-TIME
APPLICATIONS
Producer
Consumer
Platform capabilities and
standards
Enhance developer experience
FinOps management
Consistent observability
Security and governance
LARGE DATA SETS
Multi
Cloud

7. Streaming Applications
• Labyrinth of streaming paths
• Many producers and
consumers
• Multi-cloud service
providers
• Polyglot of programming
languages
• Complex event mesh

8. Governance Challenges
Increase
throughput
Avoid
common
pitfalls
Producer:
1. buffer.memory
2. batch.size
3. compression.type
4. linger.ms
5. retries
Consumer:
1. fetch.min.bytes
2. fetch.max.wait.ms
3. max.poll.interval.ms
4. max.poll.records
5. enable.auto.commit

9. Initial Approach
Documentation
Best practices
Training
Workshops

10. Initial Approach
Scale and efficiency?

11. Application Governance
Introduction of Application / Client Governance Gates
Application Observability
Performance Testing
Resiliency Testing
Client Config Linting
Client Optimization Release to Prod
Release to Prod

12. Application Governance
Introduction of Application / Client Governance Gates
Application Observability
Performance Testing
Resiliency Testing
Client Config Linting
Client Optimization Release to Prod
Release to Prod

13. Part 2
Implementation
Challenges and Opportunities
13

14. Challenges
• Thousands of producers/consumers
• 170+ client configurations
How can we help producer/consumer teams configure client apps
properly?

15. How can we convert our experiences to a tool that can help users?
Analyze

16. Rule Engine & Linting API
What feedback can we provide?
• Rule/Policy engine
• Decouple decision making logic from business code
• Maintain the rules/policies centrally as code
• Serve as a linting API to client application to provide
feedback

17. How do we integrate with the client applications?
Demo

18. What’s next?
What about the parameters that have range value?
What’s the best value for them?
Integration Details

19. Looking Forward
How to find the relationship between the parameters and the throughput?
Performance Testing
• Cost / expensive
• Too many teams

20. Part 3
Machine Learning
Continuous Learning and Optimization
20

21. How to use the model to figure out the best value of parameter
Inferring Insights using Machine Learning Models

22. Where should we start?
Use Model to Find the Better Result

23. How to find the best possible performance and the corresponding parameters
Determine the Starting Point

24. Is the model’s prediction perfect?
Inference

25. Determine the optimal values based on performance testing
Performance Testing

26. How to deal with changes and improve the model
Default allowed values
Rule base, security/best practice
Get start value to do further tuning
Inference to find better values
Much less performance testing
The optimal values of client configurations
Fine-Tuning Stages

27. Self-Learning Cycle
Keep adding new data
Keep removing obsolete data
Keep identifying the new factors
Keep improving the ML model
ML Model Optimization

28. Summary
Application Observability
Performance Testing
Resiliency Testing
Client Config Linting
Client Optimization

29. Thank You!
Kafka Summit London 2023