Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2MZXaaI. Bernd Ruecker demonstrates how the new generation of lightweight and highly-scalable state machines ease the implementation of long running services. He shares how to handle complex logic and flows which require proper reactions on failures, timeouts and compensating actions and provides guidance backed by code examples to illustrate alternative approaches. Filmed at qconnewyork.com. Bernd Ruecker is co-founder and developer advocate at Camunda. Previously, he has helped automating highly scalable core workflows at global companies including T-Mobile, Lufthansa, Zalando. He is currently focused on new workflow automation paradigms that fit into modern architectures around distributed systems, microservices, domain-driven design, event-driven architecture and reactive systems.

1. Complex event flows in
distributed systems
@berndruecker With thoughts from http://flowing.io
@berndruecker | @martinschimak

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
event-flow-distributed-systems

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. 3 common hypotheses I check today:
# Events decrease coupling
# Orchestration needs to be avoided
# Workflow engines are painful

5. Berlin, Germany
bernd.ruecker@camunda.com
@berndruecker
Bernd Ruecker
Co-founder and
Developer Advocate of
Camunda

7. Simplified example:
dash button
Photo by 0xF2, available under Creative Commons BY-ND 2.0
license. https://www.flickr.com/photos/0xf2/29873149904/

8. Three steps…

9. Who is involved? Some bounded contexts…
Checkout
Payment
Inventory
Shipment

10. (Micro-)services
Checkout
Payment
Inventory
Shipment

11. Autonomous (micro-)services
Checkout
Payment
Inventory
Shipment
Dedicated Application Processes
Dedicated infrastructure
Dedicated Development Teams

12. Events decrease coupling

13. Example
Checkout
Payment
Inventory
Shipment
The button blinks if we can
ship within 24 hours

14. Request/response: temporal coupling
Checkout
Payment
Inventory
Shipment
Request
Response
The button blinks if we can
ship within 24 hours

15. Temporal decoupling with events and read models
Checkout
Payment
Inventory
Shipment
Good
Stored
Read
Model
Good
Fetched
The button blinks if we can
ship within 24 hours
*Events are facts about what happened (in the past)

16. Events can decrease coupling*
*e.g. decentral data-management, read models,
extract cross-cutting aspects

17. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched

18. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched

19. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html

20. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html

21. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html

22. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
Fetch the goods
before the
payment

23. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
Fetch the goods
before the
payment
Customers can
pay via invoice
…

24. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
Fetch the goods
before the
payment

25. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Fetch the goods
before the
payment
Goods
fetched
Order
placed
Payment
received
Goods
shipped

26. Photo by born1945, available under Creative Commons BY 2.0 license.

27. Order
Extract the end-to-end responsibility
Checkout
Payment
Inventory
Shipment
*Commands have an intent about
what needs to happen in the future
Order
placed
Payment
received
Retrieve
payment

28. Commands help to avoid (complex)
peer-to-peer event chains

29. Orchestration needs to be avoided

30. Smart ESB-like middleware
Checkout
Payment
Inventory
Shipment
Order
Order
placed
Payment
received
Good
fetched
Good
shipped

31. Dumb pipes
Checkout
Payment
Inventory
Shipment
Order
Smart endpoints
and dumb pipes
Martin Fowler

32. Danger of god services?
Checkout
Order
A few
smart god services
tell
anemic CRUD services
what to do
Sam Newmann
Payment
Inventory
Shipment

33. Danger of god services?
Checkout
Payment
Inventory
Shipment
Order
A few
smart god services
tell
anemic CRUD services
what to do
Sam Newmann

34. A god service is only created
by bad API design!

35. Example
Order Payment
Retrieve
Payment

36. Example
Order Payment
Credit
Card
Retrieve
Payment

37. Example
Order Payment
Credit
Card
Retrieve
Payment
Rejected

38. Example
Order Payment
Client of dumb endpoints easily become a god services.
If the credit
card was
rejected, the
customer can
provide new
details
Credit
Card
Retrieve
Payment
Rejected
Rejected

39. Payment
failed
Who is responsible to deal with problems?
Order Payment
If the credit
card was
rejected, the
customer can
provide new
details
Credit
Card
Retrieve
Payment
Rejected
Payment
received

40. Payment
failed
Long-running execution
Order Payment
Clients of smart endpoints remains lean.
If the credit
card was
rejected, the
customer can
provide new
details
Credit
Card
Retrieve
Payment
Rejected
Payment
received
Smart endpoints are
potentially long-running

41. Persist thing
(Entity, Document, Actor, …)
State machine or
workflow engine
Typical
concerns
DIY = effort,
accidental
complexity
Scheduling, Versioning,
operating, visibility,
scalability, …
Handling
State

42. Workflow engines are painful
Complex, proprietary, heavyweight, central, developer adverse, …

43. Avoid the wrong tools!
Death by properties panel
Low-code is great!
(You can get rid
of your developers!)
Complex, proprietary, heavyweight, central, developer adverse, …

44. Workflow engines,
state machines
It is
relevant
in modern
architectures

45. CADENCE
Silicon valley
has recognized
Workflow engines,
state machines

47. CADENCE
Workflow engines,
state machines

48. CADENCE
also at scale
Workflow engines,
state machines

49. Zeebe.io

50. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
What do I mean by
„leightweight?“

51. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Build engine
in one line of
code
(using in-
memory H2)

52. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Define flow
e.g. in Java
DSL

53. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Define flow
e.g. in Java
DSL

54. BPMN
Business Process
Model and Notation
ISO Standard

55. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
We can attach
code…

56. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
…that is
called when
workflow
instances pass
through

57. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Start
instances

58. Payment
Now you have a state machine!

59. Payment
Easy to handle time

60. martin.schimak@plexiti.com
@martinschimak

61. Synchronous communication
REST
Order Payment
Credit
Card

62. Synchronous communication
REST
Order Payment
Credit
Card

63. Synchronous communication
REST
Order Payment
Credit
Card

64. Synchronous communication
REST
Order Payment
Credit
Card
Stateful
Retry

65. Distributed
systems

66. It is impossible to
differentiate certain
failure scenarios.
Independant of
communication style!
Service
Provider
Client

67. Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST

68. Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST

69. Distributed
systems

70. Distributed transactions using compensation *
Compensation

71. Relaxed consistency
Temporarily
inconsistent
state
But eventually
consistent
No Isolation
(as in ACID)

72. https://thenewstack.io/5-workflow-automation-use-cases-you-might-not-have-considered/

73. <= milliseconds seconds minutes, weeks, …
Business
IT
Business processes
automation
Distributed
Transactions
Orchestration
Communication in
distributed systems
long runningalways short running short running, but
potentially long running
Use cases for workflow automation

74. Biz Dev
improve
communication
improve
communication
Ops

75. Biz Dev
Leverage
state machine &
workflow engine
Living
documentation
Visibility in
testing
improve
communication
improve
communication
Ops

76. Visual HTML reports for test cases

77. Biz Dev
Leverage
state machine &
workflow engine
Living
documentation
Visibility in
testing
Understand and discuss
business processes
Evaluate optimizations
in-sync with
implementation
improve
communication
improve
communication
Ops

78. Living documentation for long-running behaviour

79. Biz Dev
Leverage
state machine &
workflow engine
Living
documentation
Visibility in
testing
Understand and discuss
business processes
Evaluate optimizations
in-sync with
implementation
improve
communication
improve
communication
Ops

80. Biz Dev
Leverage
state machine &
workflow engine
Living
documentation
Visibility in
testing
Operate with visibility
and context
Understand and discuss
business processes
Evaluate optimizations
in-sync with
implementation
improve
communication
improve
communication
Ops

81. Proper
Operations
Visibility + Context

82. Before mapping processes
explicitly with BPMN, the truth was
buried in the code and nobody
knew what was going on.
Jimmy Floyd, 24 Hour Fitnesse
„

83. Workflows live inside service boundaries

84. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

85. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

86. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

87. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

88. …
Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

89. Lightweight workflow engines are
great* – don‘t DIY
*e.g. enabling potentially long-running services, solving hard
developer problems, can run decentralized

90. Sales-Order & Order-Fulfillment
via Camunda
for every order worldwide
(Q2 2017: 22,2 Mio)

91. Code example & live demo
InventoryPaymentOrder ShippingCheckout Monitor
https://github.com/flowing/flowing-retail/
Human
Tasks
H2 H2

92. # Events decrease coupling: sometimes
read-models, but no complex peer-to-peer event chains!
# Orchestration needs to be avoided: sometimes
no ESB, smart endpoints/dumb pipes, important capabilities need a home
# Workflow engines are painful: some of them
lightweight engines are easy to use and can run decentralized,
they solve hard developer problems, don‘t DIY

93. Thank you!

94. bernd.ruecker@camunda.com
@berndruecker
https://bernd-ruecker.com
https://blog.bernd-ruecker.com
https://github.com/flowing
https://www.infoq.com/articles/events-
workflow-automation
With thoughts from http://flowing.io
@berndruecker | @martinschimak
Contact:
Slides:
Blog:
Code:
https://www.infoworld.com/article/3254777/
application-development/
3-common-pitfalls-of-microservices-
integrationand-how-to-avoid-them.html
https://thenewstack.io/5-workflow-automation-
use-cases-you-might-not-have-considered/

95. Watch the video with slide synchronization on
InfoQ.com!
https://www.infoq.com/presentations/event-
flow-distributed-systems