At StampedeCon 2014, Alex Miller (Cognitect) presented "Datomic – A Modern Database."
Datomic is a distributed database designed to run on next-generation cloud architectures. Datomic stores facts and retractions using a flexible schema, consistent transactions, and a logic-based query language. The focus on facts over time gives you the ability to look at the state of the database at any point in time and traverse your transactional data in many ways.
We’ll take a tour of the Datomic data model, transactions, query language, and architecture to highlight some of the unique attributes of Datomic and why it is an ideal modern database.
4. A database that
reconsiders...
• Immutable data model
• Flexible, extensible schemas
• Importance of time in our data
• Transactions and queries as data
• Deployment for the cloud
• Storage as a service
16. Entities + Attributes
• Entities - identity (no "type")
• Similar to: nodes in a graph, rows in rdbms
• Attributes
• Similar to: edges in a graph, cols in rdbms
• Relation from entities to values, or
• Relation from entities to entities
23. Alternative Views
Structure View
row datoms sharing a common E
column datoms sharing a common A
document traversal of attributes
graph traversal of reference attributes
25. Transactions
• Set of facts (assertions or retractions)
• Applied at a point in time
26. Transactions as Data
• Defined as data (not INSERT/UPDATE/
DELETE strings)
• Transactions are *also* entities!
• You can query them - for when they
happened and what datoms they include
• And add new facts about them!
27. Datoms in a
Transaction
;; E A V Tx op
[21005 :name "Stuart" 1000 true]
[21005 :likes 3299 1000 true]
[21005 :likes 1730 1022 true]
[21005 :likes 3299 1022 false]
29. Constraints
• Schema constraints enforced on attributes
• Transaction functions
• From old db value to new db value
• Enforce arbitrary constraints
• Can reject transactions
33. Server
Indexing
Trans-
actions
Query
App Process
I/O
App
Strings
DDL + DML
Result Sets
Storage
cache
monolithic server
Storage Service
App Process
D Peer Lib
b,c,ea,d,e a,b,d
D Transactor
Indexing
Trans-
actions
Query
Cache
App
Data
Data
Data
segments
Live
Index
Data
Segments
Data Segments
peer, transactor, storage
57. [:find ?customer ?email
:in $cust $emp
:where [$cust ?customer :email ?email]
[$emp _ :email ?email]]
“Find me the customers who are
also employees.”
Join across dbs
(d/q query custDb empDb)
implicit
join
58. [:find ?customer ?email
:in $cust $emp
:where [$cust ?customer :email ?email]
[$emp _ :email ?email]]
“Find me the customers who are
also employees.”
Join across dbs
(d/q query custDb empDb)
data patterns can be led
by database names
59. Travel Through Time
• Database *now*
• Database *last week*
• Database *if I added some transactions*
60. Views of a database
name semantics supports
(default) current state
what is the current
situation?
.asOf state at point in past
how were things in the
past?
.since state since point in past
how have things
changed?
tx report
before / after / change view of a
tx
automated
event response
.with state with proposed additions
what would happen if
we did X?
.history timeless view of all history anything!
62. Query Engine is Local
• Local query engine, local cache
• If working set is in memory, everything is FAST
• Reads do not require any transactor interaction
• Use your own functions in the query
63. Extension with custom fns
[:find ?customer ?product
:where [?customer :shipAddress ?addr]
[?addr :zip ?zip]
[?product :product/weight ?weight]
[?product :product/price ?price]
[(Shipping/estimate ?zip ?weight) ?shipCost]
[(<= ?price ?shipCost)]]
“Find me the customer/product
combinations where the shipping cost
dominates the product cost.”
predicate
64. Extension with custom fns
[:find ?customer ?product
:where [?customer :shipAddress ?addr]
[?addr :zip ?zip]
[?product :product/weight ?weight]
[?product :product/price ?price]
[(Shipping/estimate ?zip ?weight) ?shipCost]
[(<= ?price ?shipCost)]]
“Find me the customer/product
combinations where the shipping cost
dominates the product cost.”
function
67. Rules
[(relatedProduct ?p1 ?p2)
[?p1 :category ?c]
[?p2 :category ?c]
[(!= ?p1 ?p2)]]
“Products are related if they
have a common category.”
head is true ...
68. Rules
[(relatedProduct ?p1 ?p2)
[?p1 :category ?c]
[?p2 :category ?c]
[(!= ?p1 ?p2)]]
“Products are related if they
have a common category.”
if body is true
69. q("[:find ?p2
:in $ %
:where (expensiveChocolate p1)
(relatedProduct p1 p2)]",
db
rules)
“Find all products related to
expensive chocolate.”
Rule inputs
rules are a kind of input
70. q("[:find ?p2
:in $ %
:where (expensiveChocolate p1)
(relatedProduct p1 p2)",
db,
rules)
“Find all products related to
expensive chocolate.”
Naming rule inputs
rule names begin
with %
71. q("[:find ?p2
:in $ %
:where (expensiveChocolate p1)
(relatedProduct p1 p2)",
db,
rules)
“Find all products related to
expensive chocolate.”
Using rule patterns
rule patterns can
appear in :where clause
72. [[(relatedProduct ?p1 ?p2)
[?p1 :category ?c]
[?p2 :category ?c]
[(!= ?p1 ?p2)]]
[(relatedProduct ?p1 ?p2)
[?o :order/item ?item1]
[?item1 :order/product ?p1]
[?o :order/item ?item2]
[?item2 :order/product ?p2]
[(!= ?p1 ?p2)]]]
“Products are related if they have the same
category, or they have appeared in the
same order.”
Implicit or
73. ;; base case
[(story-comment ?story ?comment)
[?story :story/title]
[?story :new/comments ?comment]]
Recursive query
for graph navigation
it is a story comment if...
74. ;; base case
[(story-comment ?story ?comment)
[?story :story/title]
[?story :new/comments ?comment]]
Recursive query
for graph navigation
... there is a story ...
75. ;; base case
[(story-comment ?story ?comment)
[?story :story/title]
[?story :new/comments ?comment]]
Recursive query
for graph navigation
... with a comment
76. ;; recursion
[(story-comment ?story ?comment)
[?parent :news/comments ?comment)
(story-comment ?story ?parent)]
Recursive query
for graph navigation
or, it is a story comment if...
77. ;; recursion
[(story-comment ?story ?comment)
[?parent :news/comments ?comment]
(story-comment ?story ?parent)]
Recursive query
for graph navigation
... it has a parent comment ...
78. ;; recursion
[(story-comment ?story ?comment)
[?parent :news/comments ?comment)
(story-comment ?story ?parent)]
Recursive query
for graph navigation
which is itself a story comment
80. Direct Index Access
• seek-datoms - walk the datoms in a
specified index between particular
transactions or dates
• entid-at - gets a fabricated entity id
based on a transaction id or date
81. Direct Log Access
• Walk the transaction log directly
• Starting from txid or point in time
87. Entities
• Retrieve entity from the database as a map
• Follow references to lazily walk the graph
• Follow references in both directions
• Use touch to retrieve all entity attributes
(def ent (d/entity dbval 17592186045417))
=> {:db/id 17592186045417}
(d/touch ent)
=> {:db/doc "Hello world",
:db/id 17592186045417}
88. Sweet Spots
• Flexible data model
• Audit trail, provenance, time
• Transactional data of record
• Horizontal query scaling
• Cloud deployment
89. Leverage other
systems
• Blobs (images, sound, movies, giant text)
• Write churn (hit counter)
• Horizontal write scaling
90. Datomic Free
Datomic Pro
Starter
Datomic Pro
Cost
Redistributable?
Number of Peers
High-Availability
Transactor
Storage Services
Memcached
Support
$0 $0 per peer
Yes No No
2 2 by license
No No Yes
Local only All All
No No Yes
Community Community Enterprise