The microservice architecture functionally decomposes an application into a set of services. Each service has its own private database that’s only accessible indirectly through the services API. Consequently, implementing queries and transactions that span multiple services is challenging. In this presentation, you will learn how to solve these distributed data management challenges using asynchronous messaging. I describe how to implement transactions using sagas, which are sequences of local transactions, coordinated using messages. You will learn how to implement queries using Command Query Responsibility Segregation (CQRS), which uses events to maintain replicas. I describe the key role that messaging plays a microservice architecture. Given at Silicon Valley Code Camp 2018

1. @crichardson
Events and commands: developing
asynchronous microservices
Chris Richardson
Founder of Eventuate.io
Founder of the original CloudFoundry.com
Author of POJOs in Action and Microservices Patterns
@crichardson
chris@chrisrichardson.net
http://learn.microservices.io
Copyright © 2018. Chris Richardson Consulting, Inc. All rights reserved

2. @crichardson
Presentation goal
Using events and commands
to implement
transactions and queries
in a microservice architecture

3. @crichardson
About Chris

4. @crichardson
About Chris
Consultant and trainer
focusing on modern
application architectures
including microservices
(http://www.chrisrichardson.net/)

5. @crichardson
About Chris
Founder of a startup that is creating
an open-source/SaaS platform
that simplifies the development of
transactional microservices
(http://eventuate.io)

6. @crichardson
About Chris
https://www.manning.com/books/microservices-patterns
40% discount with code
ctwsvcc18

7. About Chris: microservices.io
Microservices pattern
language
Articles
Code examples
Microservices Assessment
Platform - coming soon

8. @crichardson
Agenda
Transactions and queries in a microservice architecture
Using event-driven microservices
From events to sagas
Implementing sagas using asynchronous request/response

9. The microservice architecture
structures
an application as a
set of loosely coupled
services

10. @crichardson
Microservices enable
continuous delivery/deployment
Process:
Continuous delivery/deployment
Organization:
Small, agile, autonomous,
cross functional teams
Architecture:
Microservice architecture
Enables
Enables
Enables
Successful
Software
Development
Services
=
testability
and
deployability
Teams own services

11. @crichardson
Each team owns one or more
services
Catalog
Service
Review
Service
Order
Service
…
Service
Catalog
Team
Review
Team
Order
Team
…
Team
Responsible
for

12. @crichardson
Microservice architecture
Browser
Mobile
Device
Store
Front UI
API
Gateway
Customer
Service
Order
Service
…
Service
Customer
Database
Order
Database
…
Database
HTML
REST
REST
Database per service

13. @crichardson
Private database
!=
private database server

14. @crichardson
Loose coupling =
encapsulated data
Order Service Customer Service
Order Database Customer Database
Order table
Customer
table
orderTotal creditLimit
availableCredit
Change schema without coordinating with other teams

15. @crichardson
But…
How to maintain data
consistency?
How to implement queries?

16. @crichardson
How to maintain data consistency?
createOrder(customerId, orderTotal)
Pre-conditions:
• customerId is valid
Post-conditions
• Order was created
• Customer.availableCredit -= orderTotal
Customer class
Invariant:
availableCredit >= 0
availableCredit <= creditLimit
Spans
services

17. @crichardson
Cannot use ACID transactions
that span services
BEGIN TRANSACTION
…
SELECT ORDER_TOTAL
FROM ORDERS WHERE CUSTOMER_ID = ?
…
SELECT CREDIT_LIMIT
FROM CUSTOMERS WHERE CUSTOMER_ID = ?
…
INSERT INTO ORDERS …
…
COMMIT TRANSACTION
Private to the
Order Service
Private to the
Customer Service
Distributed transactions

18. @crichardson
2PC is not an option
Guarantees consistency
BUT
2PC coordinator is a single point of failure
Chatty: at least O(4n) messages, with retries O(n^2)
Reduced throughput due to locks
Not supported by many NoSQL databases (or message brokers)
CAP theorem 2PC impacts availability
….

19. @crichardson
How to implement queries?

20. @crichardson
Find a customer and their
orders
SELECT *
FROM CUSTOMER c, ORDER o
WHERE
c.id = o.ID
AND c.id = ?
Customer
Service
Order Service
…. is no longer easy

21. @crichardson
API Composition pattern
Customer Service
Customer
…
Order Service
Order
…
API Gateway
GET /customers/id?expand=orders
GET /customer/id GET /orders?customerId=id
FK query!

22. @crichardson
Find recent, valuable
customers
SELECT *
FROM CUSTOMER c, ORDER o
WHERE
c.id = o.ID
AND o.ORDER_TOTAL > 100000
AND o.STATE = 'SHIPPED'
AND c.CREATION_DATE > ?
Customer
Service
Order Service
…. is even more difficult!

23. @crichardson
API Composition would be
inefficient
1 + N strategy:
Fetch recent customers
Iterate through customers fetching their shipped orders
Lots of round trips high-latency
Alternative strategy:
Fetch recent customers
Fetch recent orders
Join
2 roundtrips but potentially large datasets inefficient

24. @crichardson
Agenda
Transactions and queries in a microservice architecture
Using event-driven microservices
From events to sagas
Implementing sagas using asynchronous request/response

25. @crichardson
Domain Event
Something that has happened that is of interest to domain
experts
For example:
Order placed, cancelled
Flight arrived, delayed, departed

26. @crichardson
DDD Domain Event
Something that has happened to an aggregate
Represented by a class
Usually immutable
Used for
Auditing - include identity of user making change
Maintaining data consistency in a distributed system
…

27. @crichardson
Using events to maintain data
consistency
Order
Service
Customer
Service
Order created
Credit Reserved
Credit Limit Exceeded
Create Order
OR
Customer
creditLimit
creditReservations
...
Order
state
total
…
create()
reserveCredit()
approve()/
reject()
Order events channel
Customer events channel

28. @crichardson
Using events to update a queryable
replica = CQRS
Order
Service
Customer
Service
Order events
Customer events
findCustomersAndOrders()
Order events channel
Customer events channel
Customer
Order
View
Service
View
Database
Replica

29. @crichardson
Query side data model
Command Query Responsibility
Segregation (CQRS)
Command side data model
Commands
Aggregate
Message broker or Event Store
Events
Queries
(Materialized)
View
Events
POST
PUT
DELETE
GET

30. @crichardson
Query side design
Message broker
or Event Store
Updater
View Updater
Service
Events
Reader
HTTP GET
Request
View Query
Service
View
Store
e.g.
MongoDB
ElasticSearch
Neo4J
update query

31. @crichardson
Queries database (type)
Command side
POST
PUT
DELETE
Aggregate
Event Store/Message Broker
Events
Query side
GET /customers/id
MongoDB
Query side
GET /orders?text=xyz
ElasticSearch
Query side
GET …
Neo4j

32. @crichardson
But how to publish events
whenever data changes?

33. @crichardson
Option #1: Event sourcing
=
Event centric approach to
business logic and persistence

34. @crichardson
Event sourcing: persists an object
as a sequence of events
Event table
Entity type
Event
id
Entity
id
Event
data
Order 902101 …OrderApproved
Order 903101 …OrderShipped
Event
type
Order 901101 …OrderCreated
Order
create()
approve()
ship()

35. @crichardson
Replay events to recreate in-
memory state
Order
state
OrderCreated(…)
OrderAccepted(…)
OrderShipped(…)
Events
Periodically snapshot to avoid loading all events

36. Replaying an event updates
aggregate
Each event = state change
Creation event initializes
aggregate
Other events performs state
change
Bank account:
State = balance
AccountDebited - reduce balance
AccountCredited - increase
balance
Aggregate
state
Event

37. @crichardson
Example customer aggregate
Money creditLimit
Map<OrderId, Money> creditReservations
Customer
List<Event> process(CreateCustomerCommand cmd) { … }
List<Event> process(ReserveCreditCommand cmd) { … }
…
void apply(CustomerCreatedEvent event) { … }
void apply(CreditReservedEvent event) { … }
…
State
Biz logic
Create events
http://eventuate.io/exampleapps.htmlUpdate state

38. @crichardson
Event sourcing guarantees:
state change
->
event is published

39. @crichardson
Event Store
Event-driven architecture using event sourcing
Order 123 Customer 456
Subscribe
Order
Service
Subscribe
Customer
Service
createOrder()
Order Created
Order Created
Customer Created
Credit Reserved
Credit Reserved
Order Approved

40. @crichardson
Preserves history of domain objects
Supports temporal queries
Built-in auditing
Other benefits of event sourcing

41. @crichardson
Evolving the schema of long-lived
events
Event store only supports PK-
based access, requires CQRS
Drawbacks of event sourcing

42. @crichardson
Option #2: Traditional
persistence + business logic
explicitly publish events

43. @crichardson
Use traditional persistence:
JPA, MyBatis, JDBC, …
Publish event
Save order
http://eventuate.io/exampleapps.html

44. @crichardson
Use a very familiar technology
stack

45. @crichardson
Atomically updating state and
publishing events
ACID
transaction
See BASE: An Acid Alternative, http://bit.ly/ebaybase
DELETE
?
Customer
Service
ORDER_ID CUSTOMER_ID TOTAL
99
CUSTOMER_CREDIT_RESERVATIONS table
101 1234
ID TYPE DATA DESTINATION
MESSAGE table
84784 CreditReserved {…} …
INSERT INSERT
Message
Publisher
QUERY
Message
Broker
Publish
Local transaction
reserveCredit()

46. @crichardson
Publishing messages inserted
into MESSAGE table
Poll the MESSAGE table
Simple
But what about latency
Use Transaction log tailing
Lower latency
But implementation is database specific
MESSAGE
Table
Message
Publisher
Message
Brokerhttp://eventuate.io/exampleapps.html

47. @crichardson
Agenda
Transactions and queries in a microservice architecture
Using event-driven microservices
From events to sagas
Implementing sagas using asynchronous request/response

48. @crichardson
From a 1987 paper

49. @crichardson
Saga
Use Sagas instead of 2PC
Distributed transaction
Service A Service B
Service A
Local
transaction
Service B
Local
transaction
Service C
Local
transaction
X Service C

50. @crichardson
Saga
Use Sagas instead of 2PC
Distributed transaction
Service A Service B
Service A
Local
transaction
Service B
Local
transaction
Service C
Local
transaction
X Service C
Message/
event
Message/
event

51. @crichardson
Collaboration using asynchronous,
broker-based messaging
Order Service
Customer
Service
….
Message broker
Guaranteed delivery ensures a saga complete when its
participants are temporarily unavailable

52. @crichardson
Saga step = a transaction
local to a service
Service
Database Message Broker
update publish message/event
Transactional DB: BEGIN … COMMIT
DDD: Aggregate
NoSQL: single ‘record’
How to
make atomic
without 2PC?

53. @crichardson
But what about rollback?
BEGIN TRANSACTION
…
UPDATE …
…
INSERT ….
…
…. BUSINESS RULE VIOLATED!!!!
…
ROLLBACK TRANSACTION
Really simple!

54. @crichardson
Rolling back sagas
Use compensating transactions
Developer must write application logic to “rollback” eventually
consistent transactions
Careful design required!

55. @crichardson
Sagas complicate API design
Synchronous API vs Asynchronous Saga
Request initiates the saga. When to send back the response?
Option #1: Send response when saga completes:
+ Response specifies the outcome
- Reduced availability
Option #2: Send response immediately after creating the saga
(recommended):
+ Improved availability
- Response does not specify the outcome. Client must poll or be notified

56. @crichardson
Revised Create Order API
createOrder()
returns id of newly created order
NOT fully validated
getOrder(id)
Called periodically by client to get outcome of validation

57. @crichardson
Minimal impact on UI
UI hides asynchronous API from the user
Saga will usually appear instantaneous (<= 100ms)
If it takes longer UI displays “processing” popup
Server can push notification to UI

58. @crichardson
Sagas are ACD
Atomicity
Saga implementation ensures that all transactions are
executed OR all are compensated
Consistency
Referential integrity within a service handled by local databases
Referential integrity across services handled by application
Durability
Durability handled by local databases

59. @crichardson
Lack of I
Anomalies because sagas are
interleaved

60. @crichardson
Anomaly: Lost update
Ti: Create
Order
Tj: Approve
Order
Ti: Cancel
Order
Saga 1
Create Order
Saga 2
Cancel order
Time
Overwrites
cancelled order

61. @crichardson
Countermeasure Transaction
Model
http://bit.ly/semantic-acid-ctm - paywall 😥
https://www.slideshare.net/chris.e.richardson/saturn-2018-managing-data-consistency-in-a-microservice-architecture-using-sagas

62. @crichardson
Agenda
Transactions and queries in a microservice architecture
Using event-driven microservices
From events to sagas
Implementing sagas using asynchronous request/response

63. @crichardson
How to sequence the saga
transactions?
After the completion of transaction Ti “something” must
decide what step to execute next
Success: which T(i+1) - branching
Failure: C(i - 1)

64. @crichardson
Choreography: distributed decision making
vs.
Orchestration: centralized decision making

65. @crichardson
Option #1: Choreography-based
coordination using events
Order
Service
Customer
Service
Order created
Credit Reserved
Credit Limit Exceeded
Create Order
OR
Customer
creditLimit
creditReservations
Order
state
total
create()
reserveCredit()
approve()/
reject()
Order events channel
Customer events channel

66. Benefits and drawbacks of
choreography
Benefits
Simple, especially when
using event sourcing
Participants are loosely
coupled
Drawbacks
Cyclic dependencies -
services listen to each
other’s events
Overloads domain objects,
e.g. Order and Customer
know too much
Events = indirect way to
make something happen
https://github.com/eventuate-examples/eventuate-examples-java-customers-and-orders

67. @crichardson
Order Service
Option #2: Orchestration-based saga
coordination
Local transaction
Order
state=PENDING
createOrder()
Customer Service
Local transaction
Customer
reserveCredit()
Order Service
Local transaction
Order
state=APPROVED
approve
order()
createOrder()
CreateOrderSaga
InvokesInvokesInvokes

68. @crichardson
A saga (orchestrator)
is a persistent object
that
tracks the state of the saga
and
invokes the participants

69. Saga orchestrator behavior
On create:
Invokes a saga participant
Persists state in database
Wait for a reply
On reply:
Load state from database
Determine which saga
participant to invoke next
Invokes saga participant
Updates its state
Persists updated state
Wait for a reply
…

70. @crichardson
Order Service
CreateOrderSaga orchestrator
Customer Service
Create Order
Customer
creditLimit
creditReservations
...
Order
state
total…
reserveCredit()
CreateOrder
Saga
OrderService
create()
create()
approve()
creditReserved()
Customer command channel
Saga reply channel

71. @crichardson
CreateOrderSaga definition
Sequence of
steps
step = (Ti, Ci)
Build command
to send
Saga’s Data

72. @crichardson
Customer Service command
handler Route command
to handler
Reserve
credit
Make reply message

73. @crichardson
Eventuate Tram Sagas
Open-source Saga orchestration framework
Currently for Java
https://github.com/eventuate-tram/eventuate-tram-sagas
https://github.com/eventuate-tram/eventuate-tram-sagas-
examples-customers-and-orders

74. Benefits and drawbacks of
orchestration
Benefits
Centralized coordination
logic is easier to understand
Reduced coupling, e.g.
Customer knows less.
Simply has API
Reduces cyclic
dependencies
Drawbacks
Risk of smart sagas
directing dumb services

75. @crichardson
Summary
Database per service is essential for loose coupling
Implement some queries using event-driven CQRS
Maintaining data consistency using sagas
Implement simple sagas using event-driven choreography
Implement complex sagas using asynchronous request/
response-based orchestration
Publish events using either event sourcing or explicit
publishing

76. @crichardson
@crichardson chris@chrisrichardson.net
http://learn.microservices.io
Questions?
40% discount with code
ctwsvcc18