Should you use REST to sew services together? Is it better to use a richer, brokered protocol? This practical talk will dig into how we piece services together in event driven systems, how we we use a distributed log to create a central, persistent narrative and what benefits we reap from doing so.
2. 2
In this talk
1. How we traditionally build services with
REST
2. How Event Driven Services are different
3. Building an Immutable, Shared Narrative.
4. Leveraging Materialized Views.
5. Relating to CQRS & Event Sourcing
6. Pulling it all together
20. 20
Data is herded into specific services
Data Services
Stateless
Services
Internet Co
21. 21
Lots of Data Services
=> Distributed Join Problem
pic
This is a problem databases find hard,
and they’re highly tuned for it!
Payments Service
Orders Service
Customers Service
Client Service
performs join
22. 22
Lots of Data Services
=> Distributed Join Problem
pic
TESTING BECOMES DIFFICULT!
Client Service
performs join
Payments Service
Orders Service
Customers Service
23. 23
In short: the high degree of “connectedness”
makes it hard to evolve independently and to scale
39. 39
1. Events are Immutable Facts
(state changes taken from the real world and journalled)
UI
Service
Purchases
Payments
Purchase
Requested
Payment
Completed
All the benefits of “Event Sourcing”
40. 40
2. Services couple only to data flows
No knowledge of contributing services (RDFC)
UI
Service
Purchases
Payments
Purchase
Requested
Payment
Completed
41. 41
Because coupling is only to Events, not
services
Purchase Requests
PurchaseRequest TopicUI
Service
Payment
Service
Stock
Service
42. 42
The architecture is “Pluggable”
Purchase Requests
PurchaseRequest TopicUI
Service
Payment
Service
Stock
Service
Fraud
Service
43. 43
Communication & State Concepts Merge
• The concept of Data & Events
become one.
• Kafka is both Event Store and
Communication Channel
44. 44
The Events form a Canonical, Shared
Narrative
Payments
Orders
Shipments Customers
Evolving state of the system over time
45. 45
Scaling is a concern of the broker, not
“upstream” services
Purchase Requests
PurchaseRequest TopicUI
Service
Payment
Service
Stock
Service
Fraud
Service
Kafka
• Linearly scalable
• Fault Tolerant
• Multi Tennant
46. 46
But there is something missing!
How do we look things up?
(i.e. the Q in CQRS)
• What is the address for this customer?
• Is this user allowed to view this stock?
• What is the contents of this user’s shopping basket?
47. 47
With REST Lookups are natural, if
Remote
UI
Service
Customer
Service
http://customers/ID/42
Go to the relevant service and ask!
48. 48
Which creates the dependency problem
Many services tightly coupled to one another
49. 49
With Event Driven we create “views” or
“projections” inside each bounded context
Projection of
data from
Outside
Context
Boundary
Shipping
Service
50. 50
Pattern 1: Local KTables
Shipping Service
Customers
Events
Customers By ID
(saved by KStreams
using RocksDB)
Kafka
51. 51
Pattern 1: Local KTables
Shipping Service
Customers
Events
Customers By ID
KTable customers =
builder.table(
CustomerId,
Customer,
“customers-topic”,
“customer-store”);
customers = streams.store(
"customers-store”…);
customer = customers.get(42)
52. 52
As we’re using a streaming engine, we
can translate into any Domain Model
(projection) we wish
Shipping Service
Customers
Events
ShippingCustomers
By CustomerName
53. 53
Combine different streams from
different services
Fraud
Service
Payment
Events
Purchase
Events
PurchasePayments
payments.join(“purchases”,..)
.groupByKey()
.reduce(newValueReducer, “PurcahsePaymentsStore”)
54. 54
Pattern 1: Local KTables
• Simple to create and query
• Local to each service so fast
• Powerful DSL for transformation
• Inbuilt high availability
• No external database
• Controlled entirely within service’s bounded
context
• KTable and GlobalKTable allow scale out
55. 55
Pattern 2(a): Queryable Interface
Fraud
Service
Kafka
Fraud Service
(Creates streaming projection)
Query via Rest Interface
User Interface
Known as queryable
state in Kafka
Context
Boundary
56. 56
Pattern 2(b): Query Service
Kafka
Bespoke View Service
(Creates streaming
projection)
Rest Interface
Fraud
Service
Context
Boundary
57. 57
Pattern 2: Exposing Materialised
Projections
• Similar to Pattern 1 but with a
“Query Interface” (using Kafka’s
Queryable State feature)
• Commonly used with UI’s
59. 59
Pattern 3: External DB
• Again similar, but heavier weight
• Use when you cannot pre-compute
the view (i.e. to do ad hoc queries)
• Trick: start with only the data
(fields) you need today. You can
always go back for more.
61. 61
Pattern 4: Hybrid
• Quite common in practice
• Allows you to break out of the
async world
• Use sparingly, but do use when
appropriate (e.g. login service)
62. 62
Benefits
• Forms a central, immutable narrative
• Communication Protocol and Event Store
become one
• Better decoupling (RDFC)
• Excellent scalability
• Several approaches for managing queries,
lightweight => heavy weight
• Event sourcing & CQRS at its core
66. 66
All order logic
Orders-by-Customer view
(KStreams + Queryable State API)
UI ServiceStream
Maintenance
Fulfillment uses local state
store, which is replicated to a
redundant instance
UI uses Orders-by-
Customer View directly
via Queryable State
History Service pushes data to
a local Elastic Search Instance
Orders
Service
Derived
View
UI joins data
Tables & Streams
Fulfillment Service
History
Service
Fully redundant
instance
OrdersProduct Custo-
mers KTables
KTables KTables
Schemas
68. 68
Discount code: kafcom17
Use the Apache Kafka community discount
code to get $50 off
www.kafka-summit.org
Kafka Summit New York: May 8
Kafka Summit San Francisco: August 28
Presented by