This document discusses an event sourcing framework called Concursus. It begins with an introduction and agenda. It then covers moving from centralized to distributed systems using event sourcing, writing events first before processing them, three types of event processing schedules, the Concursus framework itself, and future directions for Concursus including stable Kafka integration and an asynchronous programming model.

1. Concursus
Event Sourcing Evolved

2. Introductions
Dominic Fox
Twitter: @dynamic_proxy
Email: dominic.fox@opencredo.com
Github: http://github.com/poetix
Concursus Github: http://github.com/opencredo/concursus

3. Agenda
1. From Presence to Presents
2. “Write First, Reason Later”
3. Three processing schedules
4. Concursus
5. Future Directions

4. Agenda
1. From Presence to Presents
2. “Write First, Reason Later”
3. Three processing schedules
4. Concursus
5. Future Directions

5. Agenda
1. From Presence to Presents
2. “Write First, Reason Later”
3. Three processing schedules
4. Concursus
5. Future Directions

6. Agenda
1. From Presence to Presents
2. “Write First, Reason Later”
3. Three processing schedules
4. Concursus
5. Future Directions

7. 1. From Presence to Presents
2. “Write First, Reason Later”
3. Three processing schedules
4. Concursus
5. Future Directions
Agenda

8. 1. From Presence to Presents
2. “Write First, Reason Later”
3. Three processing schedules
4. Concursus
5. Future Directions
Agenda

9. "I have told people over and over and over again, don't write a
CQRS framework…I can basically guarantee you that it will be
abandonware within one year, like every other one has become.”
- Greg Young, inventor of the term “CQRS”
Why Concursus?

10. Agenda
1. From Presence to Presents
2. “Write First, Reason Later”
3. Three processing schedules
4. Concursus
5. Future Directions

11. From Presence to Presents

12. From Presence to Presents
Aggregate services together into a single container or integrate
them through a single enterprise service bus...
Manage all of our data through a single relational database
schema with global constraints...
Use distributed locks and transactions to make a distributed
system behave as if it were one single system with a global
transactional semantics...

13. From Presence to Presents

14. From Presence to Presents

15. From Presence to Presents

16. From Presence to Presents
“Here the Microservice can become an escape route from reality. Within each Microservice, we can
live on a safe island of determinism and strong consistency — an island where we can live happily
under the illusion that time and the present is absolute.
However, as soon as we exit the boundary of the Microservice we enter a wild ocean of non-
determinism—the world of distributed systems, which is a very different world. You have probably
heard that building distributed systems is hard. It is. That being said it is also the world that gives
us solutions for resilience, elasticity, isolation amongst others. At this point, what we need to do is
not to run back to the monolith, but instead learn how to apply and use the right set of principles,
abstractions and tools in order to manage it.”
- Jonas Bonér, Reactive Microservices Architecture, p. 28-29

17. From Presence to Presents

18. Agenda
1. From Presence to Presents
2. “Write First, Reason Later”
3. Three processing schedules
4. Concursus
5. Future Directions

19. “Write First, Reason Later”

20. “Write First, Reason Later”

21. “Write First, Reason Later”
Scene from Endgame, by Samuel Beckett

22. “Write First, Reason Later”

23. “Write First, Reason Later”

24. Agenda
1. From Presence to Presents
2. “Write First, Reason Later”
3. Three processing schedules
4. Concursus
5. Future Directions

25. Three Processing Schedules
1.Transient
2.Durable
3.Persistent

26. Three Processing Schedules
1.Transient
2.Durable
3.Persistent

27. Three Processing Schedules
1.Transient
2.Durable
3.Persistent

28. Three Processing Schedules
1.Transient - what happens
2.Durable - what’s happening
3.Persistent - what happened

29. Agenda
1. From Presence to Presents
2. “Write First, Reason Later”
3. Three processing schedules
4. Concursus
5. Future Directions

30. Domain Model: Events

31. aggregateType: lightbulb
aggregateId: 69016fb5-1d69-
4a34-910b-f8ff5c702ad9
eventTimestamp: 2016-03-31T10:31:17.981Z
parameters: { “wattage”: 60 }
Domain Model: Events

32. aggregateType: lightbulb
aggregateId: 69016fb5-1d69-
4a34-910b-f8ff5c702ad9
eventTimestamp: 2016-03-
31T10:36:42.171Z
parameters: { “location”: “hallway”}
Domain Model: Events

33. aggregateType: lightbulb
aggregateId: 69016fb5-1d69-
4a34-910b-f8ff5c702ad9
eventTimestamp: 2016-03-31T10:36:42.171Z
processingTimestamp: 2016-03-31T10:36:48.3904Z
parameters: { “location”: “hallway”}
Domain Model: Events

34. Domain Model: Summary
Every Event occurs to an Aggregate, identified by its type and id.
Every Event has an eventTimestamp, generated by the source of
the event.
An Event History is a log of Events, ordered by
eventTimestamp, with an additional processingTimestamp
which records when the Event was captured.

35. Network
Event
sources
Event processors
Events arrive:
• Partitioned
• Interleaved
• Out-of-order
Processing Model: Ordering

36. Log is:
• Partitioned by aggregate id
• Ordered by event timestamp
Processing Model: Ordering

37. CREATE TABLE IF NOT EXISTS concursus.Event (
aggregateType text,
aggregateId text,
eventTimestamp timestamp,
streamId text,
processingId timeuuid,
name text,
version text,
parameters map<text, text>,
characteristics int,
PRIMARY KEY((aggregateType, aggregateId), eventTimestamp,
streamId)
) WITH CLUSTERING ORDER BY (eventTimestamp DESC);
Cassandra Schema

38. Cassandra
Event Store
RabbitMQ Topic
Downstream
processingLog
events
Publish
events
Cassandra & AMQP

39. Cassandra
Event Store
RabbitMQ Topic
Downstream
processing
out-of-order events
ordered query results
Cassandra & AMQP

40. Cassandra
Event Store
Kafka Topic
Downstream
processing
Event store
listener
Publish
events
Log
events
Cassandra & Kafka

41. Processing Model: Summary
Events arrive partitioned, interleaved and out-of-order.
Events are sorted into event histories by aggregate type and id.
Events are sorted within event histories by event timestamp, not
processing timestamp.
Event consumers need to take into account the possibility that an
event history may be incomplete at the time it is read – consider
using a watermark to give incoming events time to “settle”.

42. Programming Model: Core Metaphor

43. Consumer<Event>
Programming Model: Core Metaphor

44. You give me a Consumer<Event>, and I send Events to it one at a time:
Emitting Events

45. I implement Consumer<Event>, and handle Events that are sent to me.
Handling Events

46. Event-handling middleware is a chain of Consumer<Event>s that transforms, routes,
persists and dispatches events. A single event submitted to this chain may be:
■ Serialised to JSON
■ Written to a message queue topic
■ Retrieved from the topic and deserialised
■ Persisted to an event store (e.g. Cassandra)
■ Published to an event handler which maintains a query-optimised view of part of the
system
■ Published to an event handler which maintains an index of aggregates by event property
values (e.g. lightbulbs by wattage)
Event-Handling Middleware

47. Java 8 Mapping

48. Java 8 Mapping

49. Java 8 Mapping

50. Querying and Replaying

51. Querying and Collecting

52. Kotlin Mapping

53. Kotlin Mapping

54. Kotlin Mapping

55. ■ Event type and middleware implementing Consumer<Event>
provide basic mechanics for creating, processing and handling events.
■ Java 8 mappings provide a convenient, type-safe method-mapping
wrapper around these mechanisms.
■ Kotlin mappings provide an alternative wrapper based on immutable
classes.
■ Other mappings are possible – Scala, Clojure, Java with POJOs…
Programming Model: Summary

56. Agenda
1. From Presence to Presents
2. “Write First, Reason Later”
3. Three processing schedules
4. Concursus
5. Future Directions

57. Future Directions
1. Stable Kafka integration

58. Future Directions
1. Stable Kafka integration
2. Avro integration

59. Future Directions
1. Stable Kafka integration
2. Avro integration
3. End-to-end async programming model

60. Questions?