In questo talk, assieme ad Andrea Gioia, Partner di Quantyca, abbiamo spiegato il ruolo di Apache Kafka e come possano supportare architetture e soluzioni 'event driven' o perchè Kafka è un'ottima scelta per fare 'event sourcing'.
5. Message driven
Strong decoupling
La necessità di delineare per ogni componente il giusto confine che possa garantire il più basso accoppiamento con gli
altri, l’isolamento e la trasparenza sul dislocamento richiede di basare le interazioni all’interno delle moderne data
platform sullo scambio asincrono di messaggi.
Broker
Message Driven & Event Sourced / quantyca.it | confluent.io
6. Inside the broker
Smart endpoints and dumb pipes
Attenzione a non spostare la complessità all’interno del broker. Le logiche di business dovrebbero essere il più possibili
contenute nelle componenti che accedono al broker non nel broker stesso.
=
Big ball of mud Spaghetti in a box
Message Driven & Event Sourced / quantyca.it | confluent.io
7. Due tipologie di broker
Service Bus e Data Bus
Message Driven & Event Sourced / quantyca.it | confluent.io
8. Due tipologie di broker
Service Bus (orchestration over choreography)
Nelle tradizionali architettura SOA il coordinamento dei servizi esposti
da ogni sistema avviene tramite un Enterprise Service Bus (ESB)
centralizzato.
L’architettura basata su ESB permette di:
1. comporre servizi esposti da sistemi legacy
2. gestire transazioni distribuite
3. creare un anticorruption layer per isolare i sistemi monolitici
4. Facilitare il test e monitoraggio dell’infrastruttura
Message Driven & Event Sourced / quantyca.it | confluent.io
9. Due tipologie di broker
Data Bus (choreography over orchestration)
Nelle moderne architettura a microservizi si rinuncia alla
centralizzazione per aumentare la scalabilità del sistema.
Ogni servizio è modellato intorna ad uno specifico dominio di business
ed è completamente autonoma dagli altri dal punto di vista di :
1. Implementazione
2. Storage
3. Deploy
L’allineamento a livello di dati tra i vari servizi avviene tramite un data
bus basato su una piattaforma di event streaming distribuita (Kafka)
Message Driven & Event Sourced / quantyca.it | confluent.io
10. Due tipologie di broker
Architettura Ibrida e API Manager
Le due architetture possono coesistere. Il layer di servizi avrà sempre
più una doppia polarità:
● da una parte servizi esposti da sistemi legacy orchestrati tramite
ESB (SOA)
● dall’altra servizi con coordinamento decentralizzato e altamente
scalabili (Microservices)
Per i consumatori di servizi questa architettura ibrida non sarà
direttamente visibile in quanto l’accesso ai servizi passerà tramite API
Gateway che medieranno l’accesso ai servizi reali
Message Driven & Event Sourced / quantyca.it | confluent.io
11. How Apache Kafka and Confluent
enable event streaming
architectures
Message Driven & Event Sourced / quantyca.it | confluent.io
12. Events
An immutable record that something as some point has happened
Message Driven & Event Sourced / quantyca.it | confluent.io
A Sale An Invoice A Trade A Customer
Experience
15. Request-Response Applications
Message Driven & Event Sourced / quantyca.it | confluent.io
App
Service
Service
Service Service
Service
Service
Service
Service
App
■ DETERMINISTIC
■ RIGID
■ TIGHT
16. Event-Driven Applications
Message Driven & Event Sourced / quantyca.it | confluent.io
App
App
Developer
APIs
Service
Service
Service
Service
Service
Streaming
Platform
■ RESPONSIVE
■ FLEXIBLE
■ EXTENSIBLE
17. Moving from Pre-Streaming to Event Driven Architectures
Request-Response vs Event-Driven: You Need Both!
Message Driven & Event Sourced / quantyca.it | confluent.io
Request-Response Event-Driven
18. Event Centric Thinking
All Your Data is a Stream of Events
Message Driven & Event Sourced / quantyca.it | confluent.io
20. Why Didn’t it Work Before?
Past solutions turned out to be insufficient
Message Driven & Event Sourced / quantyca.it | confluent.io
Message-Oriented Middleware
No persistence
Single point of failure
Not fault tolerant
Cannot order messages
Cannot process messages in flight
Order of magnitude lower throughput
No “replay” functionality
EAI & ESBs
Not event-oriented
Fragile and bespoke
Weak transformation
capabilities
ETL
Slow and batch oriented
Point-to-point, not
publish & subscribe
Not a true infrastructure
platform
21. The World Has Changed
Apache Kafka is at the core of these industry trends and, ultimately,
the digitalisation of every business
Message Driven & Event Sourced / quantyca.it | confluent.io
Microservices Mobile Machine
Learning
Internet of
Things
22. Message Driven & Event Sourced / quantyca.it | confluent.io
The Event Streaming Platform
We saw that picture before, what’s different now?
■ APACHE KAFKA CAN STORE EVENTS
Kafka can store events for months, years, or indefinitely if useful. The limit is the amount of physical
storage available on Kafka brokers. Events are stored in order on a per partition basis and they can be
replayed and reconsumed (in order) whenever necessary.
■ EVENTS ARE STORED IN ORDER (PER PARTITION)
Order of events is preserved and the ability to operate on a ‘stream’ level, also via Kafka Streams APIs
or KSQL, provides a powerful way to simplify data integration, data cleansing, data transformations
tasks as well as use Kafka to exchange and publish entire datasets. With Kafka, events are first class
citizens, but also the full history and state is available to applications (it’s like having a database and a
messaging system at the same time).
■ KAFKA CAN EASILY OPERATE A COMPANY SCALE
Scalability, fault-tolerance, elasticity, and multi-tenancy are built-in capability.
23. Data Streaming and Maturity Stages
Message Driven & Event Sourced / quantyca.it | confluent.io
Value
Maturity (Investment & time)
2
Enterprise
Streaming Pilot /
Early Production
LOB(s) Pilot(s); Small
teams experimenting;
pub/sub / integration.
→ 1-3 use cases quickly
moved into Production -
but fragmented.
Pub + Sub Store Process
5
All data in the organization
managed through a single
Streaming Platform.
→ Digital natives / digital
pure players - probably using
Machine Learning & AI
(Relational databases -
redundant).
Central Nervous
System
1
Developer
Interest
Developer downloads
Kafka & experiments
(15 mins on laptop).
Pre-Streaming
Legacy systems.
Batch processes;
→ Complex
→ Slow / Silo’d
→ Expensive
4
Global
Streaming
Streaming Platform
managing majority of
mission critical data
processes, globally, with
multi-datacenter replication
across on-prem and hybrid
clouds. In parallel with
other Big Data
infrastructure.
3
SLA Ready,
Integrated
Streaming
Multiple mission critical
use cases in production,
with; scale, DR & SLAs.
→ Streaming clearly
delivering business
value, with C-suite
visibility.
Projects
Platform
24. Designing Event-Driven Systems
Message Driven & Event Sourced / quantyca.it | confluent.io
In this book Ben explains how service-based architectures and stream processing tools
such as Apache Kafka can help you build business-critical systems.
● Understand why replayable logs such as Kafka provide a backbone for both service
communication and shared datasets
● Explore how event collaboration and event sourcing patterns increase safety and recoverability
with functional, event-driven approaches
● Apply patterns including Event Sourcing and CQRS, and how to build multi-team systems with
microservices and SOA using patterns such as “inside out databases” and “event streams as a
source of truth”
● Build service ecosystems that blend event-driven and request-driven interfaces using a
replayable log and Kafka's Streams API
● Scale beyond individual teams into larger, department- and company-sized architectures, using
event streams as a source of truth
The book is available for free in PDF from the Confluent website!
https://www.confluent.io/designing-event-driven-systems
25. Message Driven & Event Sourced / quantyca.it | confluent.io
Royal Bank of Canada (RBC)
We saw that picture before, what’s different now?
https://www.confluent.io/customers/rbc
■ 16 Million Clients
■ 35 Countries
■ 30+ Use-cases
■ 50+ apps
■ 10+ lines of businesses
“Kafka transforms even the most basic of initiatives. Adoption of Kafka at
RBC has been massive and organic. Within the first six weeks after our
launch of Kafka, we had 37 teams asking to use Kafka for various
projects and initiatives.”
-- Kerry Joel, Senior Director,
Product Innovation, Data and Analytics
Digital
Marketi
ng
Securit
y
Consumer
Credit
Services
SaaS
Corporate
Real Estate
Investor
Services
Treasury
Services
….
FraudData
Wareho
use
Microservices
https://www.youtube.com/watch?v=WTxmHHJcHRc
26. The Streaming Platform
Technical Capabilities
Message Driven & Event Sourced / quantyca.it | confluent.io
Publish &
Subscribe
Store Process
28. In conclusione
Elementi chiave
1. Service integration: esb, microservizi, api manager
2. Service portfolio
3. Competence portfolio
4. Piano di migrazione evolutivo
Prossimi passi
1. Scalabilità
2. Governance non solo a livello dati ma anche a livello servizi
Message Driven & Event Sourced / quantyca.it | confluent.io
29. Corso Milano, 45 / 20900 Monza (MB)
T. +39 039 9000 210 / F. +39 039 9000 211 / @ info@quantyca.it
www.quantyca.it
