2. Background
▌ Landmark would be one of the top 50 largest software companies
▌ Scientific and Engineering software for Energy sector
Exploration
Construction
Production
▌ Neftex
Exploration
3.
4. What kind of developer are you?
▌ Experienced with Java (since 1.1 which is ~1998)
▌ Python hacker (what other kind is there?)
▌ Javascript noob (!! How hard can it be? Typescript, Uglify)
▌ Fortran deny all (literally don’t tell anyone)
▌ C++ dangerous (enough to be)
▌ C# not much (linux?)
▌ Wrote much UI and Server Architecture
6. Happy Developers?
▌ Modern Programming Languages like Java and Python
Developers became happy
They write more code
Thing Why we liked it? What Happened?
3G/4G
Languages
(Java, C#, Python, C++)
Manage memory,
Powerful APIs, UI
toolkits
We wrote lots of code
IDEs
(Xemacs, JCreator,
Eclipse, IntelliJ,
VisualStudio)
It make our code
more colourful and
we could write more
We wrote lots of code
Frameworks
(RCP, Struts, JSF,
Django, ASP, etc. etc.)
Do more stuff,
Organise things
We wrote lots of code and
it was more efficient
Incremental
Programs
(OSGi, Jigsaw, etc.)
Our massive
programs still worked
We wrote lots of code and
it didn’t all have to fit in
memory!!
7. Introducing the Monolith
▌ Single Tier Software Application
▌ Horizontal Approach to Features
▌ Modularity is Code-Based
▌ Examples:
Scientific Desktop Application
Some Types of Server Application
− Thick client
− Multiple web page single server
8. Some problems I have hit with…
▌ Developers working horizontally, less efficiently
▌ Encapsulation of modules – entropy increases
▌ Multiple APIs doing the same task
CORBA/RMI/JMS
Swing/SWT/JavaFX
DOM+SAX/Xpath/Castor/Gson/Jackson
▌ Ability to scale in various directions (scale cube)
▌ And… Deployment / Testing / Understanding / Agility
Despite all this
Powerful,
Well used and understood,
Profitable
Monolithic, or nearly monolithic, software
9. Some I think which can be okay with…
▌Testing
Well defined interactions
Limited set of interoperable features
▌OSGi Services – one monolith multiple services
▌Developer ownership, interoperability
▌Cost
Monolithic, or nearly monolithic, software
10. Summary
▌ We have lots and lots and lots of code
▌ We find it hard to change it
▌ Customers/Users get
releases infrequently
more defects
lower scale solutions
…depressed
11. Service Oriented Architecture (SOA)
▌ Application is a collection of loosely coupled services
▌ One service
Logically represents a business activity with a specified outcome.
Self-contained.
black box for its consumers.
May consist of other underlying services
▌ Idea been around in one form or another since early 2000’s
µServices are one type of SOA
12. µServices
▌ Fine grained and light weight
▌ Modularity is enforced, each component easy to understand, develop
and test
▌ Small autonomous development teams
Vertical development rather than horizontal
Less for one developer to deal with / understand / have to compile
▌ IMPORTANT Services can scale independently and on the fly
▌ Technology choices weakly linked and easy to change
15. This is all very well but…
▌ How do we apply it to scientific algorithms?
▌ How will they scale?
Different algorithms have different requirements
Scale out in the cloud
▌ What kind of cloud? We are looking at DC/OS, Kubernetes and native:
AWS, Azure, Google Cloud etc.
▌ What about Big Data?
16. Case Study – Automatic Lithostratigraphy
▌ Python Research Code
Machine learning
Parallelizable algorithm each python process can work on one part of the problem
prediction
▌ Java µService using JSON-REST and Kafka deployed in a Jetty Server to a docker container
▌ Large amounts of data but highly parallel each run being only O(10)xO(100,000), for instance
9 1D datasets of size 25,000.
▌ Scale in:
Y - (web service front end and data splitter) + (analysis)
X - Many (analysis) processes
Z – No need to scale
17.
18. Standard Python Project Containing
R&D Code
Module for exposing running one facet
of the code by Kafka message
(competitive consumer)
A readme / getting started and other docs are
contained in the project
Jenkins build runs the tests and
deploys the python consumer to the
cluster as a docker image
19. Python client
µservice code using JSON-REST Spring
Boot, etc
Self contained set of unit tests and resources
Separate build which
• Downloads dependencies for developer
• Builds on Jenkins a ‘fat’ jar
• Deploys to DCOS
The build is deployed to docker image for
running on the cluster
A readme / getting started and other docs are
contained in the project
20. 2ms
▌ Binary artifacts can be published and consumed = modularity old skool
E.g. IO packages
▌ µservice design patterns
E.g. Aggregator
Bit of an side note: Artifactory and Aggregation
20ms
20ms 20ms
2ms
= 62ms
2ms
20ms
2ms
= 28ms
2ms
Aggregator