Axa Assurance Maroc - Insurer Innovation Award 2024
Zero to scaleable in ten minutes
1. Zero to scaleable
in ten minutes
Less than obvious tools & tips for writing software that lasts
Matt Walters
@mateodelnorte
mattwalters5@gmail.com
2. What do you mean… ‘scaleable’?
As businesses grow, their understanding of
problems they’re solving also grows
Scaleable = easy to understand, maintain, and add to
During that growth, software changes. Good
software is easiest to change, while still getting
things done.
3. Combining standard, available tools, and repeating
simple, predictable patterns during development will
keep your code understandable, maintainable, and
easily improved.
Our toolkit will be:
• gitslave
• tmuxinator
• rabbitmq
• servicebus
*and some very simple patterns
How do we scale an architecture?
4. gitslave
Why have a monolith, when you can have a
metarepo?
> man gits
Gitslave Home Page: <http://gitslave.sf.net>
NAME
gits - The git slave repository tool for multi-repository management
SYNOPSIS
gits [-p|--parallel COUNT] [-v|--verbose]+ [--quiet] [--help] [--version] [-n|--no-pager]
[--paginate] [--eval-args] [--exclude SLAVE-REGEXP] [--keep-going] [--no-commit]
[--no-hide] [--no-progress] [--no-master] [--with-ifpresent|--just-ifpresent] SUBCOMMAND
[ARGS]...
OVERVIEW
gits is a program that assists in assembling a meta-project from a number of individual
git repositories which operate (when using gits) as if they were one git repository
instead of many, similar to the way that CVS works by default and svn (v1.5) can be
coerced to work. Some of these individual git repositories may be part of other meta-
projects as well.
5. gitslave
Why have a monolith, when you can have a
metarepo?
gitslave allows you to have all of your code in one location,
like a monolithic app, while allowing fine grained updates
of each of your solution’s component projects.
one ‘gits pull origin master’ gets all new changes, for all
related projects / repositories, while you can still perform
fine grained operations on any particular repo.
7. gitslave
Why have a monolith, when you can have a
metarepo?
> cat .gitignore
/denormalizer/
/my-service-one/
/my-service-two/
/my-site/
8. gitslave
Why have a monolith, when you can have a
metarepo?
> cat .gitslave
"../denormalizer" “denormalizer"
“../my-service-one“ “my-service-one“
"../my-service-two" “my-service-two“
“../my-site" “my-site"
15. Neat whats next?
We’ve talked about where you’ll house your project, and
the different sub-projects within it.
(hint: one gitslave metarepo and n project repos)
Let’s talk a bit about messaging.
And we’ve talked about how you’ll stare at it while it’s
on your screen.
(hint: tmuxinator)
22. Why messaging?
Your processes need an interface
Shared Spaghetti
Doable, but usually unclear which process ‘owns’ database
structure and data. Can lead to maintenance hell.
27. Why messaging?
Your system gets a safety net
Queues
?x5Added benefit:
Planned human
intervention for
errors.
28. rabbitmq
messaging that just works
• direct send to queue
• fanout / topic routing
• highly available
• highly performant
• used in financial exchanges,
industrial applications and more
• open source
• free
30. servicebus
super simple messaging in node
• direct send
• pub / sub / fanout / topic -routing
• simple to set up
• highly performant
• used in financial exchanges,
online advertising and more
• open source
• free
• perfect for creating microservices!
32. servicebus
super simple send+listen messaging in node
// process 1
var bus = require('servicebus').bus();
bus.listen('my.event', function (event) {
console.log(event);
});
// process 2
var bus = require('servicebus').bus();
setInterval(function () {
bus.send('my.event', { my: 'event' });
}, 1000);
33. servicebus
super simple pub+sub messaging in node
// process 1
var bus = require('servicebus').bus();
bus.subscribe('my.event', function (event) {
console.log(event);
});
// process 2
var bus = require('servicebus').bus();
setInterval(function () {
bus.publish('my.event', { my: 'event' });
}, 1000);
// process 3
34. (micro)servicebus
some simple patterns get us going
// order-svc index.js
const bus = require(‘servicebus').bus();
const create = require(‘./lib/create’);
bus.listen(‘order.create', function (event) {
create(event);
});
35. (micro)servicebus
some simple patterns get us going
// order-svc index.js
const bus = require(‘servicebus’).bus();
const retry = require(‘servicebus-retry’);
bus.use(retry({
store: new retry.RedisStore({
host: process.env.REDIS_HOST,
port: process.env.REDIS_PORT
})
}));
const create = require(‘./lib/create’);
bus.listen(‘order.create', function (event) {
create(event, (err) => {
if (err) return event.handle.reject();
event.handle.ack();
});
});
36. (micro)servicebus
some simple patterns get us going
// order-svc refactored bus to ./lib/bus.js
const bus = require(‘servicebus’).bus();
const retry = require(‘servicebus-retry’);
bus.use(retry({
store: new retry.RedisStore({
host: process.env.REDIS_HOST,
port: process.env.REDIS_PORT
})
}));
module.exports = bus;
37. (micro)servicebus
some simple patterns get us going
// order-svc index.js
const bus = require(‘./bus’);
// moved bus initialization to own module
const create = require(‘./lib/create’);
bus.listen(‘order.create', function (event) {
create(event, (err) => {
if (err) return event.handle.reject();
event.handle.ack();
});
});
38. (micro)servicebus
some simple patterns get us going
// order-svc index.js
const bus = require(‘./bus’);
// moved bus initialization to own module
const create = require(‘./lib/create’);
bus.listen(‘order.create', function (event) {
create(event, (err, order) => {
if (err) return event.handle.reject();
bus.publish(‘order.created’, order, () => {
event.handle.ack();
});
});
});
service publishes to the world when it’s done!
39. commands tell services
when an actor wants an action
client send commands to instruct a service to do work
commands are sent async, fire and forget
commands are sent directives: order.create
web app order-svc
order.create!
40. events tell the world
when you’re done
services publish when done performing work
services publish messages to any services that wish to listen
events are sent past-tense: order.created
order-svc fulfillment-svc
order.created!
order.created!
41. (micro)servicebus
some simple patterns get us going
// order-svc index.js
const bus = require(‘./bus’);
// moved bus initialization to own module
const create = require(‘./lib/create’);
bus.listen(‘order.create', function (event) {
create(event, (err, order) => {
if (err) return event.handle.reject();
bus.publish(‘order.created’, order, () => {
event.handle.ack();
});
});
});
42. (micro)servicebus
some simple patterns get us going
// order-svc index.js
const bus = require(‘./bus’);
// moved bus initialization to own module
const cancel = require(‘./lib/cancel’);
const create = require(‘./lib/create’);
bus.listen(‘order.create', function (event) {
create(event, (err, order) => {
if (err) return event.handle.reject();
bus.publish(‘order.created’, order);
bus.publish(‘order.created’, order, () => {
event.handle.ack();
});
});
});
bus.listen(‘order.cancel', function (event) {
cancel(event, (err, order) => {
if (err) return event.handle.reject();
bus.publish(‘order.canceled’, order);
bus.publish(‘order.created’, order, () => {
event.handle.ack();
});
});
code buildup
increases
cognitive load
43. (micro)servicebus
some simple patterns get us going
// order-svc index.js
const bus = require(‘./bus’);
// moved bus initialization to own module
require(‘./lib/handlers/orderCanceled’);
require(‘./lib/handlers/orderCreated’);
// moved handlers to their own modules,
// hiding their complexity from other code
44. (micro)servicebus
some simple patterns get us going
// order-svc index.js
const bus = require(‘./bus’);
const log = require(‘llog’);
const registerHandler = require(‘servicebus-register-handlers’);
registerHandlers({
bus,
handleError: function handleError (msg, err) {
log.error(`error handling ${msg.type}: ${err}. rejecting message`
+ `w/cid ${msg.cid} and correlationId ${this.correlationId}.`);
log.fatal(err);
msg.handle.reject(() => {
bus.close();
process.exit(1);
});
},
path: ‘./lib/handlers', // handlers are now one listen or subscribe
per file
queuePrefix: 'order-svc',
});
46. Let’s recap!
‘cause there was a lot!
1. gitslave lets you structure a project like a monolith,
but have the convenience of many repositories
2. tmuxinator gives you a head’s up view of your system,
all running locally
3. messaging provides a durable, convenient interface
between your processes
4. rabbitmq + servicebus enable easy send+listen & pub
+sub messaging patterns between processes
5. processes send commands and publish events
6. simple, expected, structure in your services keeps
cognitive load low, and projects easy to maintain
47. Thanks!
We’ll cover more on event-driven
architecture, microsvc, CQRS, and
more in a future talk!
48. Zero to scaleable
in ten minutes
Matt Walters
github & twitter: @mateodelnorte
email: mattwalters5@gmail.com
https://github.com/tmuxinator/tmuxinator
rabbitmq.com
http://gitslave.sourceforge.net/
https://www.npmjs.com/package/servicebus