This document discusses using Erlang for data operations involving multiple relational databases with different schemas located in different locations. Erlang is well-suited for this due to its concurrency, scalability, and ability to act as "glue" between systems. An approach is described using Erlang thin clients to enforce and maintain the relational data model across databases while global state resides in a traditional database. Asynchronous message passing via RabbitMQ is used to route processing jobs between agents located at different sites.
Advantages of Hiring UIUX Design Service Providers for Your Business
Erlang for data ops
1. Erlang for Data Ops
a db-centric design with Erlang
Warning: trendy people may be offended... (contains refs to relational databases, perl)
@mnacos
2. specs
● multiple relational dbs why Erlang?
● feed processing ● read the book(s), saw the
movie, been to meetups
● multiple locations
● good for concurrency,
● multiple schemas services, scalability, etc.
● automated ● can't stand middleware
● soft realtime (esp. java “frameworks”)
● interoperable
● extensible
3. approach
● good old-fashioned
relational modelling
● model escapes the
RDBMS
● Erlang as 'glue'
(erl is perl for systems)
● Erlang thin-clients enforce
and maintain the model
● global state resides in a
traditional database (logic)
controlling db schema snippet
● ACID is good – eventual
consistency across sites
4. workflow
Hub discovers / allocates work
Agents do feed processing
Agents submit messages to hub
Relational-friendly message fmt
Messages routed via rabbitmq
Each site has its own mailbox
Erl consumer applies operations
Principles:
http for synchronous
amqp for asynchronous
agents, hubs and consumers
have types...
5. message format
"t":[ single transaction
{ "d":{"virtualdb1":"tracking data"},
"r":{"public":"mycases"},
"k":[
{"company":"6678928"} Transactions such as these
], are packaged in amqp
"z":null messages with appropriate
routing keys
},
{ "d":{"virtualdb1":"tracking data"},
"r":{"public":"mycases"},
"k":[
{"company":"6678928"}, key part
{"casenumber":"9513"}
],
"z":[
{"dateregistered":"2010-09-10"},
{"location":"LONDON"}, payload
{"statuscode":"ABCD"},
{"sum":"3983.00"}
] Each element of this transaction is
} declarative i.e. a logical assertion
] ... we use UPSERTs
13. thoughts ideas
● not easy ● dynamic tuple introspection
● it's crazy ● java/scala client libraries
● i'd do it again
● erlang messes with your mind
links
http://www.rabbitmq.com/erlang-client-user-guide.html
http://github.com/mochi/mochiweb
http://code.google.com/p/log4erl/
http://github.com/mnacos/epg
@mnacos