Reactive Microservice Architecture with Groovy and Grails

Reactive Oriented Architecture with
Grails and Groovy
By Steve Pember
CTO, ThirdChannel
© 2014 SpringOne 2GX. All rights reserved. Do not distribute without permission.
@svpember
Welcome everyone to Reactive Oriented Architecture with Grails and Groovy.
!
My name is Steve, I work for a Boston-based startup called ThirdChannel, and I’ve been drinking coffee literally all day.
Let’s talk about Architecture

We’ve got a journey ahead of us folks, over an hour.
I’ll stop periodically for questions.
Last session of the day, I know it’s a haul. We’re almost there.

Agenda:
• 1. Monolithic Apps Will Ruin You
• 2. Service Oriented Architecture and Microservices
• 3. Reactive Applications
• 4. Practical Development (Caveats)
• 5. Case Studies (Including 3C)
THIRDCHANNEL
3
While I will be talking about Reactive Applications with Groovy, there’s a good deal of information surrounding the approach that should be covered first.
From a high level, I’d group these points into the following topics:
1. Monolithic Apps are Evil and will Ruin You
2. Service Oriented and Microservice Architecture, specifically their benefits and when you would or wouldn’t want to use them
3.

Have you ever...
Gone through a major refactoring of your
application?
THIRDCHANNEL 5

Have you ever...
Gone on an ‘Archeological Expedition’ in
the codebase to add a new feature?
THIRDCHANNEL 6

Have you ever...
Thought “Why are there so many #@$!%
tables in this database?!”
THIRDCHANNEL 7
Have you ever been distraught at the number of joins needed to fetch a batch of records?

Have you ever...
Dreaded executing the test suite?
THIRDCHANNEL 8
-Who here has worked on a major project where the unit tests took longer than 15 minutes to run?
-30?
- Did you ever notice that developers tend to ignore running the tests when that happens? How was build stability?
- On the flip side, has anyone worked on a major project that had effectively NO tests?

These are all symptoms of a Monolithic application. If you laughed, I’m assuming that the answers were a ‘yes’.

-A monolithic app is: Single, logical executable or unit, often comprised of 3 components for a web app: the View layer (CSS/HTML/JS), the middle tier
(contains business logic) and the data storage.
-For example, here’s an example of an e-comm app: product browsing, catalog management, order processing, shopping cart

Monoliths Feel Natural
• Contains Everything
THIRDCHANNEL
12
If you’re just starting out with building an application, a Monolith feels natural
!
The Monolith Contains all application features and all logic necessary for handling a request within a single process.

THIRDCHANNEL 13
Your developers can generally run ANNNND test the entire app at once, on their laptops

• Minimum Viable Product can be reached quickly
THIRDCHANNEL
14
— Perhaps their most attractive feature, is that a development team - even a very small one - can reach the Minimum Viable Product very quickly.

• Minimum Viable Product can be reached quickly
• Single Deployment
THIRDCHANNEL
15
There’s only a single artifact to deploy, which makes deployment and DevOps incredibly easy! … (he says sarcastically)

Monolithic apps scale horizontally by deploying whole new instances of the monolith, probably behind a load
balancer.
!
This is wasteful, as we scale the entire application rather than just the portions that require it.
For example -> if we’re receiving a flood of requests to the Order Processing, we don’t need to scale the whole
thing
!

I generally regard these single, logical units as an Evil piece of software.
Like I said, it feels like the right thing to do… and they tempt you with how easy it is to in the beginning.
Why?

You start out strong, adding controllers here, domain objects, etc.
You can be blinded by early gains toward your first release that you become shortsighted and aren’t thinking towards the future.

As those Domains grow, you start adding relations all over the place.
Can you really look at a class or schema diagram like this and really think, “Yeah, that’s awesome. Nailed it.”?
-btw, this was the largest I could find on Google images. I’m sure we’ve all seen the massive versions of this that cover multiple pages, taped to the wall.
- If you ever have to tape to the wall, done something wrong

The Complexity of your application will soon become enormous.
!
Any gains you think you may have made at the beginning of the project will not continue, particularly once you start to attract users.
..And grow your team.

Scaling Development is
hard
Scaling Development is hard work

The ability to add new features, fix bugs, or resolve technical debt cannot scale with the size of the application. Throwing more developers at it also will not
help, they’ll just get in each others’ way.
- One of the best passive-aggressive statements I’ve seen
- Passive aggressively scrawled on the wall: You can’t make a baby in 1 month with 9 women.

-And trying to refactor anything? Forget it.
-I’ve seen it happen plenty of times: Your company will start out by creating a team to reimplement a feature or rebuild an application. Sounds good?
Eventually, this team will start taking too long. Soon enough, the managers will be screaming for updates or new features on the old app. So, you end up
with *two* teams, one maintaining the old app, while the other continues the rebuild AANNND maintains feature parity with the old system.

THIRDCHANNEL 25
As the size of your codebase increases, the computational complexity will exponentially increase and will have adverse effects on maintenance. This has
been known for some time; there’s been papers written on the subject. Although it’s difficult to accurately measure.

It can also just be downright miserable. Has anyone here ever opened their project, looked at the maze of controllers or domains - that seem to just
explode out at you when opening the folder - and… just give an exasperated sigh?
Toiling away on a project which is always growing… always broken… and never seems to get fixed. You work on a small piece all day and nothing ever
seems to get better? …This can have an adverse effect on developer morale and motivation.

This is such a misguided architecture, it makes you wonder how exactly we collectively ended up here.

I blame it largely at the feet of the large, popular web frameworks, e.g. Rails, Django, Grails.
!
Don’t get me wrong, they have their uses.
And yes, I see the … irony… in complaining about them while at a framework conference
!

But they’re touted as if they’re the magic cure for building your app or product; that your company should be
based entirely within a framework. You hear folks say, “Oh, what are you using in your startup?” “Why, We’re a
Ruby on Rails shop, obviously!”.
!I
argue that your framework choice is largely irrelevant (unless it’s Grails ;)! ). When people ask you “what
technology are you using?” the answer should be something like “Ugh, well, that’s difficult”.

Architecture Choice is More
Important Than Any
Framework

Architecture Choice is More
Important Than Any
Framework*
*And we all know that only JVM languages should even be
considered

Questions so far?
• 1. Steve really hates Monolithic apps
!
• 2. Monolith Development will not scale
!
• 3. Architecture Choice > Frameworks
THIRDCHANNEL
32
So, to recap:
!!
Any questions?

-Let’s take our anecdotal design, an e-commerce store.

With SOA, one would break up each feature into an individual component or service, and setup communication between each service over some direct
service calls. Eg SOAP or REST.
-Note that these calls are traditionally Synchronous over http, so it’s imperative that each node return as quickly as possible

This architecture keeps keeps the individual features easier to manage and keep track of, which reduces overall complexity dramatically as we can
examine each service in isolation, rather than one big code base

However, before I go into why that’s so important: SOA, specifically, is not without its own issues.
!
First, having that web of interconnected services can grow out of control. Having to configure service nodes so that each knows where the others are
located in the network can be cumbersome to manage.

ESB
THIRDCHANNEL 38
Furthermore, anecdotally, organizations tend to treat SOA differently and have different definitions for what, exactly, SOA is.
-In many cases, Enterprise Service Buses (or ESBs) have arisen to deal with complicated traffic. An ESB acts as a piece of middleware that ‘intelligently’
routes traffic to the appropriate server. Which sounds useful, except This adds an additional layer of complexity to an already complex approach,
especially if you’re rolling your own ESB.
-Often, I’ve seen clients that would hook up several monoliths together and call it SOA
-It’s terrible when these all come together.

The Rise of
Microservices
Given those pain points, I think that leads right into talking about Microservices, which is a buzzword you certainly couldn’t have escaped hewing about
after that keynote last night. And every other talk here at SpringOne

Microservices
• Distillation of SOA
THIRDCHANNEL
40
So what exactly is a Microservice architecture?
!I
firmly believe Microservices were born out of a desire to be more specific when we talk about Service Oriented Architecture.
!

If you look at what people are writing about the approach, you can see that it is clearly an attempt to be more focused about using services.
And, certainly, how we manage the teams working on them
!

Microservices
• Single Context
THIRDCHANNEL
42
A Microservice should be concerned with only one ‘context’. This helps to avoid having each service become a monolith on its own

Back to this example, here we’ve split up each of the specific components into their own bounded context.
!I
say “bounded context” because that’s an important metric. You may hear folks say “ A microservice needs to be under 1000ish total lines of code”. Or
something like that.
!
Which is Ridiculous. A microservice should surround a particular bounded context within your application. Meaning that it is the responsible for a
particular section of contiguous business logic… it’s the authoritative source on one ‘thing’.

Microservices
• Single Context
• Smart Services / Dumb Pipes
THIRDCHANNEL
44
-A Microservice Architecture emphasizes that communication between services be ‘dumb’.
-As opposed to ESBs, which can contain complex routing and even business logic
- Rather, keep all business logic contained within the relevant Service

Back to this example. Each service should communicate with each other via ‘dumb’ (as in, single-minded) methods.
For example, we could use straight up HTTP to talk between services.
Be “of the web”. Use Http Resources (HATEOAS anyone?) between services

Or, Use Message Brokers
Or, Preferably - in my opinion - Communicate between your services using an asynchronous Message Broker and Message Queues
!
There are several excellent ones that you can use. We could send quite a bit of time just talking about RabbitMQ

Message Brokers generally follow this pattern:
-Messages are routed through an EXCHANGE in to one or many queues.
-Messages are then plucked off the queue by an attached, waiting Consumer. Or, the first available if multiple are
attached.

-we’ve broken the features into individual applications and spun up one node per each
-connected the users’ cart to a queue, and our order processing into another queue
!
- the quickest win in doing just this is that now our application can place items in a users cart and take orders
*asynchronously*. Imagine a flurry of users ordering from you and you’re unable to take their money fast
enough.
- here, the order is placed on the queue and your users go back to browsing or purchasing more. This frees up
request threads in the front-end for other users as you’re not blocking on the billing component, which may
require expensive credit card operations.

Microservices
• Single Context
• RESTful Communications
THIRDCHANNEL
50
In a distributed system. One of the most ridiculous things you can do is try and share state across services.
Thus a core component of a microservices architecture is to NEVER share state

Attempting to keep shared state across distributed services WILL lead to madness.
You can certainly have mutable state *within* a service, but trying to sync that is just ridiculous.
As we were talking about before with the simple communication strategy: make REST a part of it

Microservices
• Single Context
• RESTful Communications
• Dedicated, Cross-Purpose Teams
THIRDCHANNEL
52

Teams in charge of a service should come from a variety of disciplines. Engineering, QA, Art & Design, Project Management, perhaps a deployment
engineer, etc.
! (If anyone laughs : Yes, I mixed Star Trek and Star Wars references.

Benefits of SOA & Microservices
Before I … eventually… get into some of the … caveats with this architecture,
Let me tempt you with why it’s great
!
Why would I want to do this?

Yay for Microservices
• It’s easier to reason about smaller codebases
THIRDCHANNEL
55
It’s dramatically easier to reason about smaller codebases

Each engineer should be able to understand and hold the majority of the service within their mind,.
!
Thus, when a defect occurs, it’s easier to know exactly where the problem code lies.
When a new feature is needed, it’s easier to know exactly where to build and how it will affect other components.

!
Maintenance and adding new features now becomes vastly easier. You’ll find that iterating on a particular service is much quicker, and it’s easier to
context shift when and if it’s time to work on another service.

Which results in rapid long-term
development
In addition, by focusing our developers into teams responsible for each service, this keeps them more focused and out of each others way. Which, again, is
a very common problem with monolithic apps. If you’re working on a monolith, how often are you getting merge conflicts?

• Easy To Scale / Deploy / Destroy
THIRDCHANNEL
59

-we’ve broken the features into individual applications and spun up one node per each
-connected the users’ cart to a queue, and our order processing into another queue

-suppose the cart and order processing start experiencing heavy load
- programatically spin up new instances to handle it
- Does everyone remember Andy Glover’s talk from yesterday? He had a thread on how rapidly iterating is the key to beating out your competitors.. and I
think, right here, is exactly what he was talking about…
-

• Easy To Scale / Destroy /Deploy
• Easy To Refactor
THIRDCHANNEL
62

Both in terms of adding new features, or removing old ones
! If we need to refactor a particular feature in a service… say, the inventory check when adding a product to a cart, I can do so without impacting the rest of
the application.
!
Further, if we decide to totally revamp our billing service (say we want to try out Ratpack), we can develop the new service quickly (as we’re starting with
something new) and slap it on the queue, again without directly impacting or interrupting the others.
!
A/B Testing: We could also compare our refactored billing service by having both old and new listen on the queue, then compare how effective the two
billing services are.

• Right Tool for the Job
THIRDCHANNEL
64
-not every service needs use the same languages, frameworks
- With an SOA or Microservice approach, each service can be constructed with the technologies that best suit its context and functionality.

-If the main product browse / web interface is best suited for a standard web framework, like Grails… perfect
-What about the shopping cart? Maybe it should be built with a Redis Data source? No sweat
-How about the Order placement? Let’s use some Spring Boot, with some Spring Integration on a Postgres database.

Plus, I firmly believe that this helps to improve the morale of your developers.
- When it’s time for a new service, people get generally excited.
-Get to work with something a bit new
-Design a new project
-There’s contentment in that

• History Is Isolated
THIRDCHANNEL
67
Because each service is kept as a separate product, it should ideally have it’s own repository. Therefore, one nice added effect is that each product now
has it’s own isolated revision history

• History Is Isolated
• Mitigates Conway’s Law
THIRDCHANNEL
68
Has anyone heard of Conway’s law?

Conway’s Law
“organizations which design systems ... are constrained
to produce designs which are copies of the
communication structures of these organizations”
-Melvin Conway, 1968
http://en.wikipedia.org/wiki/Conway's_law
THIRDCHANNEL 69
It’s a tongue-in-cheek quote from programmer Melvin Conway from the 60’s. His law states that the structure of a computer system reflects the social
structure of the company that made it. While something of a joke, it does have some truth to it.

QA
UX / Design
Engineers
DBAs
Front End Service
Internal Tools Services
Order Management Service
THIRDCHANNEL 70
Slide of different groups of people (Designers, Front-End Engineers, Back-end engineers, DBAs, Dev Ops, QA) (color coded)
vs
image of different service teams with different color coded people in them (e.g. front end team with a couple designers and no DBAs; order-management
system with 2 DBAs and no designers)

Save Money!
!
Probably the biggest advantage of all of this is that in the end, it should save you time, headaches and money
!

Save Money
THIRDCHANNEL
Fewer Resources
&
More Efficient Developers
72
If any of you perked up or are skeptical, I don’t blame you.
Mentioned this to a coworker, and he said “All I see is that you want to pay Amazon a bunch more money for these additional machines”

Borrowed this slide from Martin Fowler’s website
!
As demand on the application grows, I can scale just the components that are needed, rather than needing to replicate the whole monolith as demand
grows

In the beginning, this may not seem like a huge advantage.
!
However, consider that as your application becomes ever larger and more complex… more resources are going to be needed to power the entire thing.
!
Which results in more machines and resources that may just be there to patch over the the small bounded context of your application which is truly
demanding the attention.
If I have 6 x demand per usual on the billing system only, I argue that it should cheaper to add 6 small billing system nodes than to have to deploy our
monolith 6 times across more expensive machines

Save Money
THIRDCHANNEL
Fewer Resources
&
More Efficient Developers
75
!

Smaller Codebases Are Faster
CodeBases

Developers are More
Expensive than Machines

Spending Less Money on Developers
and Machines

Use that money to buy developers
beer
developers love beer

There are certainly some downsides of the micro service approach, but we’ll get into that later.
!
Generally though, I hope you agree that SOA and Microservices do a decent job of fighting the Monolith…

… but we can do even better. Squeeze out even more power from our resources.

Questions?
• 1. Microservices are the natural enemy of Monoliths
• 2. Small Code is Fast Code
• 3. Save Money
THIRDCHANNEL
82

CAP Theorem
Next, I want to diver a little and discuss something called the CAP Theorem

Cap Theorem
• (C)onsistency
• (A)vailability
• (P)artition Tolerance
THIRDCHANNEL
84

Cap Theorem
• (C)onsistency
• (A)vailability
!
• Pick 2
THIRDCHANNEL
85
And the CAP Theorem says that in a distributed application, your system can only fulfill 2 of those attributes

Cap Theorem
• (C)onsistency
• (A)vailability
!
• Pick 2
• Theorem From Paper Which Proved ‘Brewer’s Conjecture’
THIRDCHANNEL
86

Originally called Brewer’s Conjecture
In 2012, this paper contained a formal proof of the Conjecture and it graduated to full-on theorem.
!

Cap Theorem
• (C)onsistency
• (A)vailability
!
• Pick 2
• Not Fixed / CAP Choices Can be at Service Level
THIRDCHANNEL
88

Cap Theorem
• (C)onsistency
• (A)vailability
!
• Pick 2
• Not Fixed / CAP Choices Can be at Service Level
• Origin of Eventual Consistency
THIRDCHANNEL
91
A little bit of trivia -> this paper also introduces the term the term “Eventual Consistency” originated

Eventual Consistency
• High Consistent System: After Update, all components see
change
• Eventually Consistent System: After Update, eventually all
components will see change
THIRDCHANNEL
92
… as it propagates throughout the system.
And that’s ok, for most things.
Again, for anything dealing with Money, you’re going to want a highly consistent system.

Architecture Choice is
More Important Than Any
Framework
Now, knowing all of that, I reiterate:
Architecture choice…

If We accept that Microservices are
Good… how do we build each Service?
How can we squeeze even more power out of our applications? How can we further increase the maintenance advantage of microservices? How can we
further reduce our resource costs?

Reactive Microservices
are the ideal Web
Architecture
And so I believe that the ideal architecture for the web is one based on Reactive Applications built on distributed microservices -> not merely SOA.
A distributed Microservice architecture, where each service follows the Reactive approach

Going Reactive
But I’ve been stalling… So, when I say ‘Reactive’, what exactly do I mean?

The term ‘Reactive’ is a bit of a buzzword…

Popularized by a company called Typesafe. They are the maintainers of the Play and Akka Frameworks,
!!

4 Reactive Principles
• 1. Event-Driven
THIRDCHANNEL
100
Principle 1: Event-Driven

Asynchronous Message Passing
Essentially, a Reactive system should be built around the concept of Asynchronous Message Passing.
Take advantage of the asynchronous features of your language or framework, and…

-Communication within the system should be done via done via asynchronous immutable events, rather than long procedural code.
-This naturally promotes highly decoupled code. Sender and recipient can be constructed without having to know - or care - about implementation details
of the others.
- I think this is a very important point… and is perhaps the most important facet of Reactive programming as the others may come naturally from adopting
this single step

NEVER BLOCK
In other words, though…

THIRDCHANNEL 104
Suppose we had an E-commerce app which included a service which created and processed orders.
In this version, we use several additional service collaborators to do things like verifying that we have inventory, charge the user, and send emails if
successful or not.
Each of those service calls could take a while (checking inventory, sending email, money), which blocks the thread while we wait for each service call. This
slows down the request not only for the end user but for any rampaging hordes trying to access our app

VS
<slow>Now, compare this to…

THIRDCHANNEL 106
Here’s that same method. Where did the rest of the code go?
First, note the ‘notify’ with a String-based event

THIRDCHANNEL 107
On order:created, we execute the first services job
Then we check the inventory for holding, and broadcast a new event based on those results

THIRDCHANNEL 108
! -Now, some of might be saying, “But Steve, that code is spread across 3 different slides!”. While, true, which of these would you rather use? Which version
The moneyService will bill the client if the inventory is good,
While the email service will respond either way.
! -(Imagine having to mock all these services as collaborators, versus simply creating events)
do you think is more decoupled? Which would be easier to test?

• 1. Event-Driven
• 2. Scalable
THIRDCHANNEL
109
Principle 2. A Reactive Application must be scalable

-The system should be able to efficiently grow based on the demand placed on it.
Should be able to replicate easily on or across multiple nodes.
!
Dynamic scaling is an optimal use of resources: can scale up quickly when needed, but can scale down when not
needed (save $)
!

Application Topology will Change
As we build, we may not know what the application topology will be at the start… and it may change in the future. Design your systems such that they:
<next>

1. Minimize Contention on
Resources

-if one thread or process is blocking access to a service or another resource, than other threads, connections, etc will have to wait. This reduces the CAP
availability, will require more resources (e.g. more machines), and will anger the user if they notice the slowdown.
-horrible contention will have a direct monetary cost as you scramble to scale your resources and keep the end user happy
-this will start to come as a natural emergence of adopting Asynchronous message passing

2. Maximize Locality of
Reference
Include -> part 1. here, build up a list
Maximize locality of reference.
This is a very complicated topic…

So much so that I couldn’t think of a snappy image for it.
! -If data is accessed frequently, it’s a good candidate for caching (Temporal)
-Keep related data close together (Spatial). Minimum number of joins to build a particular record
!-
This takes on a different, sinister, twist when working with Microservices, which I’ll get into in a bit

• 1. Event-Driven
• 2. Scalable
• 3. Resilient
THIRDCHANNEL
116
Principle # 3, Be Resilient

Embrace Failure
In fact, *embrace* that catastrophic failures will most assuredly occur.

Have the mindset that at any given moment, your code is going to break. Think to yourself: How coupled is my code? How dependent on other services,
domain objects, third party services, etc is my application?
-How many of your systems are held together by metaphorical duct tape?
What happens if any one of those go down?
The worst thing you can do is know that catastrophe is coming or how it *could* come and to do nothing about it.

-A Reactive application is resilient to failure. If one service node breaks down, the others should be able to take up the slack. If one feature goes down, the
others should still operate.
-Your system should be able to suffer damage and still operate.
-For example, in the ecomm example, if the order placement feature goes down.. while your team is scrambling to fix, the end user should still be able to
browse the products and add items to the cart.
-They should just see a message that says “Sorry, order placement will be right back”…
-or better yet, take the order, send a note to the customer, and process the orders later once the system is back online

Independent Things Fail
Independently
Your resilient mantra should be “Independent things fail independently”

Occurs naturally with Async and
Microservices
Design your systems and services to be as loosely coupled as possible.
Using Async messages and microservices helps this behavior to arise naturally

• 1. Event-Driven
• 2. Scalable
• 3. Resilient
• 4. Responsive
THIRDCHANNEL
122
Principle #4, Responsive

Your application, in general, should respond to user interaction as quickly as possible. Thus, each service should handle events quickly.
!
A more ‘responsive’ application is a pleasing experience to your end user. The faster your application responds to their input, the less time they sit staring
at your app, and the happier they’ll be.
!
TODO: Add Amazon delay -> cost
Several years ago, an engineer with Amazon named Greg Linden revealed that the company ran experiment where they intentionally slowed down site
performance
- 100ms delay resulted in 1 percent drop in sales.
!

The 4 rules of Reactive Applications are…
-Sorry, I watched Fight Club a week or so ago.
As a result, I’ve got a bit of an anti consumerist streak going.
… he says while giving his presentation on a Mac laptop

1. NEVER BLOCK
!
2. SHARE NO STATE
!
3. INDEPENDENT THINGS
FAIL INDEPENDENTLY
!
4. BE FAST

No Single ‘Correct ‘ Reactive
Architecture

Rather, Reactive is a mindset or a state of being. It’s a goal that you need to orient your thinking around when
you are designing or architecting your individual service nodes and your distributed architecture

The people at Typesafe wrote up this thinking into something they call the ‘Reactive Manifesto’.
After this talk, if you like what I’m saying, feel free to go and sign it.

“An Application must be Reactive from top to
Bottom”
- The Reactive Manifesto
THIRDCHANNEL 130

So, going back to the original e-commerce example…
- how to go reactive?

First of course, is to break up our application into individual components. We could build an API to communicate
between services via synchronous HTTP…

Never Block!
But remember, never block!

Route State-less Messages
Asynchronously
Each service should send and respond via the message broker by passing messages asynchronously. (rule #1)
Each message should be entirely stateless (rule #2).

- A Reactive system will emerge from Reactive services.
- Once you reach the point where messages are asynchronously bouncing around your system, you’re going to
sit back and just whistle. Unless you’re me. I can’t whistle.

Reactive Questions?
• Reactive Is a Mindset
!
• 4 Rules
!
• Reactive Services Emerge into a Reactive Distributed App
THIRDCHANNEL
136

Practical Development
Anyway, onto a more interesting topic.

138
THIRDCHANNEL
Development
• Compose Services From libraries
! -an organization will tend to standardize the deployments and internals of new services. This is generally a good thing, but can lead to a great amount of
by nature, working with micro services have an amazing amount of code reuse.
code reuse and boilerplate

THIRDCHANNEL 139
Slide of legos?
- Thus, extract the common code into sets of shared, tested, libraries
- Compose every service out of these shared libraries, so that when working on each service you only have to reason about the custom code

Development
• Architect / Design Team
THIRDCHANNEL
140

As mentioned earlier, need an architect to keep everyone on task and following the Reactive microservice
approach.

This can be an authoritative committee as well, if that’s too much power in one person’s hands
But keep in mind that everyone in that committee is *going* to have an opinion

Development
• Empower your Teams / Make them Responsible
THIRDCHANNEL
143

Development
• Embrace Eventual Consistency
THIRDCHANNEL
145

Embrace Eventual Consistency
The concept that when a change is made to an object within the distributed system, that each service or database may not be consistent with that change,
but they eventually will be. This can be a difficult thing for people to accept, but for most objects I argue that’s OK.
! -Amazon misses an order

Development
• Embrace Eventual Consistency
• Be worried about Reference Locality
THIRDCHANNEL
147

Seriously, be concerned about
Locality of Reference
Each service should be an authority on some section of your data
In general, other services should query the authoritative server when necessary (Thus maintaining spatial locality)
- However, each separate service will need to be aware of the domain objects. Each Service will have a different understanding - or ‘projection’ - into the
particular Domain

In an ideal environment, each system would be completely separate.
-However, there will always be some conceptual overlap. For example, let’s say we have a system where <describe system>.
-Red lines are anchors between data localities
- Here, users’ uuid is a shared natural key which acts as each services’ anchor to understanding the concept of a user. In our main user service
understands a plethora of personal information
- Our order history system also understands the user, but simply maintains the order history joined with the user id

Same chart from before, but another service, email. This service also conceptualizes the user, but with a different subset of information. Namely, the email
address.
- Email address is stored in the user information service, but is *also* used extensively by the email service. Which one should be the authoritative
source? The email service uses it more (and thus has higher Temporal locality), but email is a key piece of information which should be managed by the
user auth system.
- Tough Choice
- One Compromise is to (unfortunately) synchronize the email address across multiple services
- Or, communication service has no concept of User’s email address, but simply blindly sends emails out. The email addresses involved must be placed
into each message received by the Communication service

Practical Development
with Groovy
The Groovy / Grails / Spring / general JVM world is filled with very interesting libraries and tools. It’s an exciting time to be building these scalable,
distributed applications.

THIRDCHANNEL 152
Akka
- I’m not smart enough to understand Actors
Actors -> relatively small implementation. Each receives messages in their ‘mailbox’ or message queue.
Walled off from you in terms of configuration and deployment: you access actors via a proxy. This means that actors can be easily distributed across
machines for dramatic increase in computational power.

THIRDCHANNEL 153
Reactor (which totally needs a logo)

Reactor is my favorite. It’s the easiest to use with the most straightforward applicability.
A Reactor instance is a wrapper around one of several different Dispatcher types (e.g. a standard Event Loop, an LMAX Disrupter Ring Buffer).
A reactor acts like a central dispatcher within your service node

The folks at Netflix have been kind enough to give us ReactiveX, a suite of various reactive tools for a variety of (primarily JVM) languages.
!

<sample rxGroovy Code>
The rxGroovy (or rxJava) libraries give you a tool called an Observable Sequence with which it’s possible to assemble reactive applications.
!
We can do simple asynchronous list processing or transformations…

…
or a publish subscribe approach
I believe it’s an extraordinarily useful tool for working with Groovy sequences in an asynchronous fashion.

Plus the documentation is well done, with some excellent diagrams showing event flow

! In my opinion, the two most useful currently are the Promise support and Asynchronous Request Handling
And, with Grails specifically, we have the built in Async Support

Promises!
!
EXpand on that promise a bit

Slightly more complicated example, using promise chaining. A few things to note:
- use of async keyword
- merge -> GOOORRMM!
-

Asynchronous request Handling, very simple case

Remember to verify that your servlet version set to 3.0 in BuildConfig.groovy

But Be Careful
It’s not all sunshine and roses (cupcakes and snowflakes?), however.
There are areas of difficulty with Reactive microservices, both in deciding whether or not to build them and the practical development.

Caveats
• Not The Right Fit For Everyone
THIRDCHANNEL
165

THIRDCHANNEL 166
-A Word press blog that your parents are building, a local restaurant site, or any general informational page…
-Probably not a good candidate for this approach
- Now, for your startup… you may be saying “I’m not yet a huge internet giant. This seems hard”.
- How much more difficult will it be when you eventually realize that a distributed approach would be a huge help?
-

Caveats
• Initial Investment is LARGE
THIRDCHANNEL
167

Caveats
• Initially Very Difficult
THIRDCHANNEL
168
This approach is an investment…

THIRDCHANNEL 169
An investment of both time and money. At first, a micro service architecture may look crazy, particularly perhaps, to your bosses. It’ll pay off dividends in
the end, but that initial investment can be cost prohibitive.

Caveats
• Make sure Everyone’s on Board
THIRDCHANNEL
170

! If someone on your team doesn’t agree or doesn’t Understand the microservices approach, it’s possible that they may create faulty services

Caveats
• Can Lead to Team Animosity
THIRDCHANNEL
172

Because teams are responsible for a service, it can lead to an us-vs-them mentality. If a service is broken, the teams relying on that service can began to
fall back to a ‘Well I can’t do any work because the User Management service is down”.
Encourage teams to cooperate. Cycle team members to other groups periodically.

Caveats
• Can Lead to Team Animosity
• Conway’s Law
THIRDCHANNEL
174
Be ever watchful for Conway’s Law. Ensure that your teams are working together when designing their services… otherwise you’ll find that certain services
are tailor-made to communicate more efficiently and directly with others.

Case Studies
In the end, though, it’s all worth it.
!
Some of the largest companies out there are demonstrating to the world that this approach is ideal for the web.

First up: our good friends Netflix
!
-make heavy use of AWS
-replicate bundles of services as demand dictates
-Netflix is big on resiliency. In order to test their resiliency, they use a custom tool called ‘Chaos Monkey’

Every weekday between 9am and 5pm (a.k.a. “Work Hours), Netflix unleashes this little adorable little guy into
their infrastructure, where he goes to work randomly destroying various services.

And the Chaos Monkey was just the first. They have a whole army of monkeys that do things like:
- simulating high latency and service degradation
- clean up unused resources
- monitor health checks to detect and alert for unhealthy instances

These services need to prove their strength and stand up to the simian army.
Or rather, this allows Netflix to find weaknesses and bugs in their services

Netflix <3’s Groovy
- Groovy is used extensively throughout Netflix’s services
- Whatever insane amount of the world’s web traffic goes through their services? That’s touching Groovy code
- the next time someone tells you that Groovy’s too young, or we need Java, or some other excuse, just point to Netflix

…Twitter is another large one

They had one of the largest, most monolithic Rails deployments in the world.
-Engineers needed to pull in Global Experts to get anything done on the codebase
-Spent more time on “Archeological Digs” or “Whale Hunting expeditions” to track down bugs or learn how
things worked than on developing new features

Turning point came during 2010 World Cup. Flurry of tweets brought twitter to its knees.
Service was unavailable, engineers worked through the night
Debated putting a vuvuzella sound effect here.

https://blog.twitter.com/2013/new-tweets-per-second-record-and-how

They shattered their monolithic application into multiple individual applications..
The size of the circles represent the number of deployment nodes of that particular application service
-I know I just spoke about Rabbit, but Twitter developed their own tool called Finagle which handles
asynchronous communication between nodes

Each application had their own small, determined, dedicated teams.

-they switched from the Ruby Virtual Machine to the JVM.
-Embraced Event-based programming from the bottom up
Went with Netty (as opposed to something like NodeJS), which is an NIO technology for the JVM

Monolithic: 200 - 300 requests /
sec / machine
!
Reactive: 10-20k request / sec /
machine

During the 2010 World Cup, Twitter hit a
record of 3283 TPS
During the 2010 World Cup, Twitter reached a record (at the time) of 3283 tweets per second before the service collapsed.

Now: average 5700 TPS
Dynamically scale to 143,199 TPS
Now, they average 5700 tweets per second… but their systems are set up so that they can dramatically scale up on demand. The current record is 143,199
tweets per second.
That’s not a hard number… that’s just the most they’ve received. They could theoretically handle more, it just hasn’t happened yet.

With 5x - 12x fewer nodes
And they’ve done all of this with 5 to 12 times fewer service nodes.
!
Not bad, yeah?

//That’s kind of mind blowing, right? Not Bad, eh?
!
-> Incidentally, JVM advocates and Rails detractors love to talk about how Twitter moved away from Rails to Java.
And while that’s true that Java would give better performance, the Architecture choice was vastly more important

About ThirdChannel
THIRDCHANNEL 196
- I should spend a bit of time explaining my company ThirdChannel. Normally, I might save it, but in order to fully understand the context it’s good to have a
bit of background on us

This is a slide from a sales deck, I apologize for the marketing speak
But basically, the problem we’re trying to solve…
<explain 3c>

About 3C
• In-Store Product Awareness
!
• Observational + Sales Data
!
• Product Intelligence & Analytics
THIRDCHANNEL
198

Another sales slide, but this gives a flavor of what our current feature set is(or 2 months ago current, actually), and how it looks on mobile

We Need Many Services
… or rather, we have many internal bounded contexts that I feel can be utilized as discrete services

(Show slides of services, adding one by one)
First, we have the web interface.
-Contains the Marketing Site, plus the basic structure of the internally facing site.

!
We started with gathering observational data (rather than empirical sales data), and this second service is still our most used context (beyond, of course,
the front-end service).
This service is responsible for receiving and providing a query-able interface for the data we gather.

To gather all of that data, we use Agents. However, each Customer could have several dozen to several hundred agents, so we need services to both help
our staff with Agent Recruitment, and Agent Management (e.g. which stores they’re currently assigned to).
The checkin server and I believe also the web interface query the agent management service to determine which stores an agent is both shown and
allowed to provide data for.

And we do pay those agents pretty well… so we need a service to track payments per some payment schedule. Each Customer may pay agents differently,
so that logic needs to be handled.
Every two weeks

And all of that, so far, is mostly just for Agents!
!
Notification Service: sends emails and tracks internal messages within the app.

And then services for pulling sales data…

as well as a bunch of additional services for aggregating data into different query able formats

And we’re still going
-And we still have more to do
- This is just what we’ve managed to work towards this year. Is each one perfect? Absolutely not. We’re iterating on each one of those services - plus a few
other very minor ones - constantly…. and rapidly, without impacting other services or areas of the business
- if we go back one slide, we’re mostly working on the analytics services right now … in addition to revamping our main front end - and we’re not impacting
the large agent management team we have at all

Our Lessons Learned
During this process there have certainly been stumbling blocks, though

#1
The absolute First Thing…

Lessons Learned
• #1 Agree on a Messaging Format
THIRDCHANNEL
211

Agree on a Messaging format
! (Image of strife) Our discussions weren’t actually like that. But, you know. not far off.
What data type are you going to use xml? json? How about format? Something already established, like HAL?
How much data should be sent? Should every service be able to customize the response it wants? What about when broadcasting general informational
updates

Lessons Learned
• #2 Build Communication Libraries
THIRDCHANNEL
214
The very second thing that should be built is a shared library (or libraries) for facilitating communications that utilizes the pre-mentioned messaging
formats

Lessons Learned
• Maintain API Documentation
THIRDCHANNEL
215
Next, make sure that - for each service - you maintain proper API documentation

This documentation about each service should cover it’s externally facing concerns.
!
Describe which events it emits and what sort of data it sends out during each event.
Describe how to communicate directly with the service, if at all. Do you talk to it directly via HTTP? Does the service respond to RPC calls over the
message queue? What sort of input does it expect?

Lessons Learned
• Remember Conway’s Law
THIRDCHANNEL
217
Keep communication up between service teams… ESPECIALLY when you’re just starting out.
!
When these communication patterns are still in the early stages, it’s very important that everyone in your entire team understand how it will work
! I’d recommend a meeting once or twice a week for service team leads to meet and discuss things like “So I’m planning on emitting this event. Any
thoughts one data you all might need?”, or “Hey, can you update the shopping cart service to also send this data along with event Y?”

Lessons Learned
• Always Think of Data Locality
THIRDCHANNEL
218

Lessons Learned
• Always Think of Data Locality
• Consider Capturing & Logging All Events / Auditing
THIRDCHANNEL
220

Reactive Microservices are the ideal
Web Architecture
The ideal framework for the web is a Reactive Microservices approach

If you want to work with Reactive Microservices, you need to adopt the mindset.

Finally… we should all strive to be Like Netflix.

Thank YOU!
And with that, I say THANK YOU.

Any Questions?:
• 1. Monolithic Apps will Ruin you
• 2. Service Oriented Architecture and Microservices
• 3. Reactive Applications
• 4. Practical Development (Caveats)
• 5. Case Studies (including 3C)
THIRDCHANNEL
228
Steve Pember
CTO, ThirdChannel
@svpember
steve@thirdchannel.com
In conclusion, this is what I attempted to go over today.
Any questions?

Image Credits
Complexity (pipes): http://www.flickr.com/photos/bhophoto/384574407/
Simple Pipeline: http://tierracos.com/our-companies/tierra-pipeline/
Highway: http://farnorthdallas.advocatemag.com/wp-content/uploads/2013/04/882349_500402383328407_539732406_o.jpg
Telephone Exchange Operator: http://fineartamerica.com/featured/telephone-exchange-1920s-granger.html
People in Queue: http://www.guardian.co.uk/money/2010/mar/23/dole-queue-jobseekers-online
Cookie Monster: http://muppet.wikia.com/wiki/Cookie_Monster_Through_the_Years
Too Many Cooks: http://www.ecommercesystems.com/featured-articles/cooks-kitchen-driving-ecommerce-business/
Resuscitate: http://www.dailymail.co.uk/health/article-2034160/Do-resuscitate-Theyre-fateful-words-meaning-doctors-wont-try-save-you-collapse-hospital.
html
Twitter Logo & app chart: https://blog.twitter.com/engineering
Modern Server Farm: http://bookriot.com/2013/03/26/book-less-libraries-and-other-contemporary-realities/
Grey World: http://upload.wikimedia.org/wikipedia/commons/d/d0/BlankMap-World-1ce.png
Monolith: http://filmessaysandarticles.wordpress.com/
Star Trek: http://www.gifbin.com/984064
Reactive Manifesto: http://www.reactivemanifesto.org/
Cheetah: http://www.livescience.com/21944-usain-bolt-vs-cheetah-animal-olympics.html
Buzzword Bingo: http://www.amsterdamadblog.com/columns/buzzword-bingo/
Businessman meditating: http://www.huffingtonpost.co.uk/2013/07/09/meditation-successful-businessmen_n_3567694.html
Chaos Monkey: http://luckyrobot.com/netflix-chaos-monkey-keeps-movies-streaming/
Darth Vader: http://www.businessinsider.com/hillary-clinton-darth-vader-poll-2014-7
Turtles: http://people.wcsu.edu/pinout/herpetology/tcarolinei/29366343.EasternBoxTurtle2.jpg,
http://www.huffingtonpost.com/2013/04/26/24-tiny-turtles-who-need-a-reality-check_n_3134793.html,
http://www.ecology.com/2012/05/23/world-turtle-day/
“Maintenance Rocks”: http://www.prlog.org/12205104-grace-hill-inc-declares-september-maintenance-appreciation-month.html
NFL Ref: http://www.mprnews.org/story/2012/09/03/news/nfl-referees
“Say No to Sharing”: http://www.cultofandroid.com/15519/sprint-starts-say-no-to-sharing-campaign-against-shared-data-plans/
“Dominoes” https://www.flickr.com/photos/louish/5611657857/sizes/l/in/photostream/
“First Day on the Internet Kid” http://knowyourmeme.com/photos/239112-first-day-on-the-internet-kid
“Tortoise and the Hare”: http://www.theguardian.com/sustainable-business/social-enterprise-blog/2014/feb/10/week
“Star Trek”: http://keyepisodes.blogspot.com/2013/10/star-trek-next-generation.html
“Darryl Issa” http://nypost.com/2014/03/05/shouting-match-erupts-during-house-hearing-on-irs-scandal/
“Fight Club” http://pyxurz.blogspot.com/2012/05/fight-club-page-4-of-7.html
“Woman looking at phone” http://remedybail.com/blog/page/4
“Guy with metal detector” http://beachurbanity.blogspot.com/2010/04/beach-metal-detector.html
“Happy dog” http://sydneespetgrooming.com/blog/
“Trekking in the desert”: http://www.wildmorocco.com/
“morpheus”: http://www.desktopexchange.net/movie-pictures/matrix-wallpapers/
“Monoliths and Microservices” http://martinfowler.com/articles/microservices.html#CharacteristicsOfAMicroserviceArchitecture
“Calvin fighting” : http://commonsenseatheism.com/?p=2214
“Sample Restaurant” : http://www.letseat.at/
“Solar panels”: http://prepare-and-protect.net/2014/02/solar-panels/

Reactive Microservice Architecture with Groovy and Grails

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (14)

Semelhante a Reactive Microservice Architecture with Groovy and Grails

Semelhante a Reactive Microservice Architecture with Groovy and Grails (20)

Mais de Steve Pember

Mais de Steve Pember (20)

Último

Último (20)

Reactive Microservice Architecture with Groovy and Grails