Guide to Append-Only Data Stores and Event Sourcing

Append-only data stores

vs

onsdag 16 oktober 13

Append-only data stores
Only add, never remove or change
Can retrieve old values


Source code
Version controlled
Keep all versions


The problem


Mutable state

id

email

jakr


name
Jan Kronquist

jan.kronquist@jayway.se

Mutable state

id

name

email

jakr

Jan Kronquist

jan.kronquist@jayway.se

UPDATE customer
SET email="jan.kronquist@jayway.com"
WHERE id="jakr"


Mutable state

id

name

email

jakr

Jan Kronquist

jan.kronquist@jayway.com

UPDATE customer
WHERE id="jakr"


Mutable state

id

name

email

jakr

Jan Kronquist


UPDATE customer
WHERE id="jakr"

I didn’t get conﬁrmation of the order last week?


Mutable state

id

name

email

jakr

Jan Kronquist


UPDATE customer
WHERE id="jakr"

I didn’t get conﬁrmation of the order last week?
Could you reset my email address?


The technical details

SELECT count(*) AS inSweden
FROM customer
WHERE email LIKE "%@jayway.se"

SELECT count(*) AS elsewhere
FROM customer
WHERE email LIKE "%@jayway.com"

total = inSweden + elsewhere


Human errors

DELETE
FROM customer
WHERE email LIKE "j%"


Scalability - the easy parts
Caching

Static content


Caching

Static content

Stateless work


Caching

Static content

Stateless work
Load balancer


Scalability - the hard parts
State + behavior


Example domain


Rock - Paper - Scissors


http://rock-paper-scissors.com/
The future Facebook of Rock Paper Scissors
Millions of users
Many games per user


Playing the game

Player A

Player B

Server

Playing the game

Player A

Server
Game 123

Player B

Playing the game

Player A

paper

Server
Game 123

Player B

Playing the game

Player A

paper

Server
Game 123
Player B: paper

Player B

Playing the game

rock
Player A

Server
Game 123
Player B: paper
Player A: rock

Player B

Playing the game

Player A

Game 123
winner: Player B
loser: Player A

Server
Game 123
Player B: paper
Player A: rock

Player B

Playing the game
GAME WON
other move
(victory)

CREATED

WAITING
any move

other move
(tie)

GAME TIED

Player A

Game 123
winner: Player B
loser: Player A

Server
Game 123
Player B: paper
Player A: rock

Player B

Entities


Entities
Player
+ id
+ name
+ email


Entities

+
+
+
+
+
+


Game
id
state
players
moves
winner
loser

Player
+ id
+ name
+ email

Entities

+
+
+
+
+
+


Game
id
state
players
moves
winner
loser

Player
+ id
+ name
+ email

Move
+ move
+ player

Actions
Create game

POST /games

Make move

POST /games/123
move=rock

Get game result

GET /games/123


Actions
Create game

POST /games

Make move

POST /games/123
move=rock

Get game result

GET /games/123

1. Form (make move)
2. Waiting
3. Game result
4. Error


Clojure data structures


Lists - singly linked
(1 2 3 4 5) (fred ethel lucy) (list 1 2 3)



Vectors - indexed access
[1 2 3]


["fred" 17 3.14 "bar"]


Vectors - indexed access
[1 2 3]

["fred" 17 3.14 "bar"]

Maps
{:name


"Jan Kronquist" :age 37}

Event Sourcing
with
Event Store


Event Sourcing
Events that have happened
Ordered in time
Source of truth


http://geteventstore.com
Runs on .NET and Mono
Free (store) or Commercial (HA)
API

€1500/year

Custom TCP (c#)
ATOM over HTTP (xml/json)

Features
Event streams, projections, generate new events (CEP)


Event example

Server


EventStore

Event example
CreateGameCommand{:game 123 :creator "player-1"}

Server


EventStore

Event example

Server

EventStore

GameCreatedEvent{:game 123 :creator "player-1"}


Event example
MakeMoveCommand{:game 123 :player "player-1" :move "rock"}

Server

EventStore

MoveMadeEvent{:game 123 :player "player-1" :move "rock"}


Event example
MakeMoveCommand{:game 123 :player "player-2" :move "paper"}

Server

EventStore

MoveMadeEvent{:game 123 :player "player-1" :move "rock"}
MoveMadeEvent{:game 123 :player "player-2" :move "paper"}
GameWonEvent{:game 123 :winner "player-2" :loser "player-1"}


Current state?
GameCreatedEvent{:game 123, :creator "player-1", :time 1}
MoveMadeEvent{:game 123, :player "player-1", :move "rock", :time 2}


Current state?

GAME WON
if victory

CREATED

WAITING
any move

other move
if tie

GAME TIED


Current state?
GameCreatedEvent

GAME WON
if victory

CREATED

WAITING
any move

other move
if tie

GAME TIED


Current state?
MoveMadeEvent

GAME WON
if victory

CREATED

WAITING
any move

other move
if tie

GAME TIED


Aggregates
Scope of consistency when handling a command

Game
+ id
+ state
+ players
+ moves
+ winner
+ loser


Move

Player

+ id

+ id

+ move

+ name

+ player

+ email

Aggregates as Event Streams

stream = game-1
version = 2


event1 GameCreatedEvent{:game 1...}
event2 MoveMadeEvent{:game 1....}


stream = game-1
version = 2


stream = game-2
version = 4

event1
event2
event3
event4


GameCreatedEvent{:game 2...}
MoveMadeEvent{:game 2....}
GameWonEvent{:game 2....}


stream = game-1
version = 2


stream = game-2
version = 4

event1
event2
event3
event4


curl "http://127.0.0.1:2113/streams/game-1"
-d @eventdata.json
-H "Content-Type:application/json"
-H "ES-ExpectedVersion: 2"


Projections


Projections

for (Event events : allEvents) {
if (event instanceof GameCreatedEvent) {
created++;
}
}
return created;


Projections


created = 2

Projections
created = 2


Projections
created = 2

(state, event) -> {
if (event instanceof GameCreatedEvent) {
return state++;
}
rerturn state;
}


Projections in Javascript
fromAll()
.when({
$init: function () {
return { created: 0, completed:0 };
},

});


fromAll()
.when({
},
GameCreatedEvent: function(s, e) {
s.created++;
return s;
},

});


fromAll()
.when({
},
s.created++;
return s;
},
GameWonEvent: function(s, e) {
s.completed++;
return s;
},

});


fromAll()
.when({
},
s.created++;
return s;
},
s.completed++;
return s;
},
GameTiedEvent: function(s, e) {
s.completed++;
return s;
}
});


fromAll()
.when({
},
s.created++;
return s;
},

});


s.completed++;
return s;
},
/projections/games-counter
{
s.completed++;
created: 15,
return s;
completed: 10
}
}

External Projections
Business logic

events


Business logic

Projections

events


events

Business logic

Projections

events

events

Available
Consistent


Commands

Queries

events

events

Available
Consistent


System with EventStore



Command
Handler



1.command


Command
Handler


1.command


Command
Handler

2.get events

Event
Store


1.command


Command
Handler

2.get events
3.add events

Event
Store


1.command


Command
Handler

2.get events
3.add events

4.update
projections

Event
Store


1.command

Command
Handler

2.get events
3.add events

5.get events
or projections

Some
service


4.update
projections

Event
Store


1.command

Command
Handler

2.get events
3.add events

4.update
projections

Event
Store

5.get events
or projections

Some
service


6.update state


1.command

Command
Handler

2.get events
3.add events

4.update
projections

Event
Store

5.get events
or projections

7.query


Some
service

6.update state

More info
Functional Programming with DDD
http://skillsmatter.com/podcast/design-architecture/ddd-functional-programming

A deep look into the Event Store
http://vimeo.com/53153270


http://www.datomic.com/
Runs on JVM

€2300

Free (3 peers, local storage) or Commercial (HA, distrib)
Features
ACID transactions, declarative query engine


Fact
Something known to have happened or existed


Fact

2004-10-01 the email of Jan was ”jan.kronquist@jayway.se”


Fact

2004-10-01 the email of Jan was ”jan.kronquist@jayway.se”
2007-01-01 the email of Jan was ”jan.kronquist@jayway.com”


Atomic fact (datom)
Entity
Attribute
Value
Time


Atomic fact (datom)
Entity

The person Jan Kronquist

Attribute

email

Value

”jan.kronquist@jayway.se”

Time

2004-10-01


Data perception

Facts

Add facts
Add more facts
Even more facts

time

Data perception

Facts

query

Add facts
Add more facts
Even more facts

time

Data perception

Facts

query

Add facts
Add more facts

query
Even more facts

time

Data perception

Facts

Add facts
Add more facts

query
Even more facts

time

Database deconstructed


Transactor
Consistency by serializing transactions


Transactor

Peer
Your code + Query engine


Transactor

Peer
Your code + Query engine

Storage service
Key/value store
Scalable reads and writes
eg DynamoDB, InﬁniSpan, Riak + ZooKeeper, SQL(!)


System with Datomic
Peer A

Transactor

Distributed


Storage

Service

System with Datomic
1.command

Peer A

Transactor

Distributed


Storage

Service

System with Datomic
1.command

Peer A

2.read and cache

Transactor

Distributed


Storage

Service

System with Datomic
1.command

Peer A
3.transact
2.read and cache

Transactor

Distributed


Storage

Service

System with Datomic
1.command

Peer A
3.transact
2.read and cache

Transactor
4.write

Distributed


Storage

Service

System with Datomic
1.command

Peer A
3.transact
2.read and cache

Transactor
4.write
5.notify

Peer B


Distributed

Storage

Service

Datomic example

Peer


Transactor

Datomic example

Transactor

Peer

[123
[123


:creator
:state

"player-1"
:created

T1]
T1]

Datomic example

Transactor

Peer

[123
[123

:creator
:state

"player-1"
:created

["player-1" :email "player@gmail.com" T0]


T1]
T1]

Datomic example

Transactor

Peer

[123
[123

:creator
:state

"player-1"
:created

T1]
T1]



Datomic example
MakeMoveCommand{:game 123 :player "player-1" :move :rock}

Transactor

Peer

[123
[123
[123_1
[123_1
[123
[123


:creator
:state
:move
:player
:moves
:state

"player-1"
:created
:rock
"player-1"
123_1
:waiting

T1]
T1]
T2]
T2]
T2]
T2]

Datomic example
MakeMoveCommand{:game 123 :player "player-2" :move :paper}

Transactor

Peer

[123
[123
[123_1
[123_1
[123
[123
[123_2
[123_2
[123
[123
[123
[123

:creator
:state
:move
:player
:moves
:state
:move
:player
:moves
:state
:winner
:loser

"player-1"
:created
:rock
"player-1"
123_1
:waiting
:paper
"player-2"
123_2
:won
"player-2"
"player-1"

T1]
T1]
T2]
T2]
T2]
T2]
T3]
T3]
T3]
T3]
T3]
T3]

Aggregates?
Only by convention
cas (compare and set)
isComponent on relation

Game
+ id
+ state
+ players
+ moves
+ winner
+ loser


Move
+ id
+ move
+ player

Query language
What to ﬁnd
Where clauses

[entity attribute value]

Implicit joins
Call arbitrary function


Query example - basic

[:find ?email
:where
[_ :email ?email]]


Query example - basic

[:find ?email
:where
[_ :email ?email]]

[["jan.kronquist@jayway.com]
["player-1@gmail.com"]
["player-2@gmail.com"]]


Query example - aggregation
[:find ?state (?count game)
:where
[?game :state ?state]]


:where

[[:created
[:waiting
[:won
[:tied


4]
1]
7]
3]]

:where

[[:created
[:waiting
[:won
[:tied

4]
1]
7]
3]]

inProgess = created + waiting


:where
fromAll()
.when({
},
s.created++;
return s;
},
s.completed++;
return s;
},
/projections/games-counter
s.completed++; {
return s;
created: 15,
}
completed: 10
});
}


[[:created
[:waiting
[:won
[:tied

4]
1]
7]
3]]

inProgess = created + waiting

Query example - join

[:find ?email (count ?game)
:where
[?player :email
?email]
[?game
:winner ?player]]


Query at any time
(q ’[:find ?email
:where
[_ :email ?email]]
db)


Query at any time
(q ’[:find ?email
:where
[_ :email ?email]]
db)

(q ’[:find ?email
:where
[_ :email ?email]]
(as-of db #inst "2013-09-02"))


Query over time
(q ’[:find ?e (count ?tx)
:where
[?e :email _ ?tx]]
db)


Query over time
:where
[?e :email _ ?tx]]
db)

:where
[?e :email _ ?tx]]
(history db))


Java API

List<Object> results =
Peer.q(someQuery, conn.db());

Future<Map> txResult = conn.transact(data_tx);


More info
http://www.datomic.com/videos.html


Comparison


Comparison
Data model
Schema evolution
Queries
Recommendation


EventStore Events
✓ Understandable by normal people
✓ Designed for integration
✓ Forces aggregate design (scalability)
✓ Capture intent
๏ Require design effort to get right
๏ Complect transaction and query


Datomic Facts
✓ Support partial information
✓ Normalization support
✓ CRUD out of the box
๏ Static schema


EventStore schema evolution
✓ New events
✓ New event attributes
✓ Event attributes rename (change deserialization)
✓ Projected state change (rebuild projection)
๏ Cross event refactoring


Datomic schema evolution
✓ New or removed attributes
✓ New or removed entity types
✓ Move attributes between entities
๏ Fact attributes rename not possible
๏ Fact attribute type change not possible


EventStore queries
✓ Projections use JavaScript
✓ Persistent
๏ Require projection


Datomic queries
✓ Declarative
✓ Logic-based
✓ Allows calling out to your own code
๏ New language to learn


When to use Event Sourcing?
Domains where events seem natural
Different teams with different models


When to use Datomic?
Data can be modelled using entities and attributes
History is or may be interesting


More details
http://www.jayway.com/author/jankronquist/
https://github.com/jankronquist
http://www.slideshare.net/jankronquist


Questions?


Guide to Append-Only Data Stores and Event Sourcing

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Recently uploaded

Recently uploaded (20)

Guide to Append-Only Data Stores and Event Sourcing