Microservices principles are revolutionizing the way applications are built, by enabling a more decoupled and decentralized approach to implementation, creating greater agility, scalability and resilience. These applications still need to be connected to one another, and to existing systems of record. Agile integration architecture brings the benefits of cloud-ready containerization to the integration space. It provides the opportunity to move from the heavily centralized ESB pattern to integration within more empowered and autonomous application teams. We look at the architectural differences in this approach compared to traditional integration, and also at how it enables more decentralized organizational structure better suited to digital transformation. You can read a more detailed paper on this approach at http://ibm.biz/AgileIntegArchPaper. This presentation was recorded for Integration Developer News (http://www.idevnews.com/) and is available here: http://ibm.biz/AgileIntegArchWebinar
6. Resilience
Minimized dependencies,
discrete failover, fail fast,
start fast
5
Maturity
§ Are you ready for a radical change in methods, skillsets,
infrastructure, operations.
§ Are you sufficiently automated (infrastructure, test, dev pipeline,
deployment etc.)
Maintenance
§ Will you be able to sustain the skillsets needed to maintain the
microservices architecture in the future?
Latency & Serialization
§ A request/response chained down a set of microservices must incur
extra latency from network hops and serialization
§ Serialization has advanced massively in recent years, but
inevitably has some contribution to CPU usage
Data sharing
§ Not all data can be split into neat independent functions. Some
things are shared, and this needs careful design
Real-time dependencies and their combined availability
§ Microservices calling other microservices synchronously need careful
consideration
§ Tends to creep, as one service builds on top of another
§ Need to move to more complex message based techniques and/or
introduce availability patterns such as circuit breaker
Manageability
§ How do you manage and monitor a vast network of microservices
§ How do you diagnose problems across a heavily distributed
landscape
How does persistence work?
§ Pessimistic versus Optimistic
§ How to handle shared objects
§ Relational / NoSQL
§ ACID / BASE / CQRS / Event Sourcing?
Considerations
Microservice
component
Microservices ApplicationMonolithic Application
Agility
Faster iteration cycles, bounded
contexts, dedicated teamsâŠ
Scalability
Elastic scalability,
workload orchestration,
cloud infrastructure
7. Application
SOA relates to enterprise service exposure *
Application ApplicationApplication
Service oriented architecture (SOA)
and microservices architecture relate to different scopes
Microservice
application
”Service
”Service”Service
”Service
Microservices relate to
application architecture
* this simple distinction can be contentious depending on your definition of SOA
Microservices vs SOA - short blog and video (10 mins)
http://ibm.biz/MicroservicesVsSoaBlog, http://ibm.biz/MicroservicesVsSoaVideoShort
Original PoV paper on microservices and in integration (~ 15 pages) http://ibm.biz/MicroservicesVsSoa
Webinar based on above paper (55 mins) http://ibm.biz/MicroservicesVsSoaFullWebinar
8. The fate of the ESB Pattern: Moving to agile integration
Part 2: Moving to lightweight, agile integration
http://ibm.biz/AgileIntegArchPaper
Containerization
Centralized
ESB
Fine-grained
integration
deployment
Application autonomy Polyglot runtimes
Decentralized
integration
ownership
Integration as a
microservice runtime
Part 1: The fate of the ESB
http://ibm.biz/FateOfTheESBPaper
more material
http://ibm.biz/AgileIntegArchLinks
10. Organizational Granularity
9
Slice along application domain/ LoB boundaries
Centralised integration architecture
Often done at dev time and at the artefact level
Occasionally some separation of runtime monitoring
e.g. accounting and chargeback
Rarely completely segregated
Only enforced by audit compliance
Agile integration architecture
Complete separation through Dev-Prod
Supports business agility
Minimises blast radius of radical change
Independent release cycles
Payroll
Supply
Chain
Warehouse
Inventory
Common utils
and formats
Version control
Payroll
Supply
Chain
Warehouse
Inventory
Payroll
Supply
Chain
Warehouse
Inventory
Common utils
and formats
Prod
Common utils
and formats
Splitting along organisational boundaries is a natural first step, initiating a more decentralised approach to
integration. Further splits along lower level application and functional boundaries can then be explored.
14. microservice application
Governance in a decentralized organisation
Flattened organizational structure
Architects/leaders/pioneers still work in a team.
They influence via âguildsâ on key focus areas
Not all microservice teams are equal, some are
pioneers, some are factories.
Multi skill teams (business user, developer,
tester, scrum leader, âŠ)
Pioneer roles (tool specialist, tool builder,
framework specialist, infrastructure specialist,
method specialist)
Full stack developers - UX, API, integration,
data, etc.)
microservice
component
microservice
component
microservice
component
Guild(s)
15. How much do you uniquely need to know for each stack?
What if we could ensure that the only unique skill was artefact creation?
Operations Operations Operations Operations Operations
Deployment Deployment Deployment Deployment Deployment
Build Build Build Build Build
Artefact
Creation
Artefact
Creation
Artefact
Creation
Artefact
Creation
Artefact
Creation
Security Security Security Security Security
Installation Installation Installation Installation Installation
Resource
allocation
Resource
allocation
Resource
allocation
Resource
allocation
Resource
allocation
16. How much do you uniquely need to know for each stack?
What if we could ensure that the only unique skill was artefact creation?
Operations Operations Operations Operations Operations
Deployment Deployment Deployment Deployment Deployment
Build Build Build Build Build
Artefact
Creation
Artefact
Creation
Artefact
Creation
Artefact
Creation
Artefact
Creation
Security Security Security Security Security
Installation Installation Installation Installation Installation
Resource
allocation
Resource
allocation
Resource
allocation
Resource
allocation
Resource
allocation
21. What capabilities does the cloud platform provide to all runtimes
Runtime
Artefacts
Resources
Security
Operations
Routing
Deployment
Delivery
Delivery
e.g. Docker build, Jenkins, Git
Resource allocation
e.g. Kubernetes, Mesos
Deployment
e.g. Kubernetes, Helm
Routing
e.g. Istio, Linkerd
Security
e.g. Kubernetes/Istio/SPIFFE
Operations
e.g. ELK stack
20
Runtime specific
Provided by platform
22. Pets Cattle
"Cattle not pets with IIB"
http://ibm.biz/CattlePetsIIB
Benefits of Cattle vs Pets
23. HA2HA1
Challenges of traditional deployment topologies
Load bal. Load bal.
P1
P1
P2
P2
Characteristics
HA pairs
Scaling manual and vertical
Defined nodes
Explicit install and configure
Explicit cold/warm HA & DR
Peak CPU licensing
Dedicated OS instances/HW
Deploy to running shared servers
Replication across DCs
Administer live shared servers
Code is only joined with the
servers at deployment.
DR1
D2
D1
DR2
D1
D2
Load bal. Load bal.
Deployment
Manager
HA
Manager
Compile/
Build
Process
Code
repository
Release
repository
HA
Manager
a b
c
System
Administration
replication
Authoring
a b
c
Monitoring
Product specific component
Product specific artefact
24. Cloud platform
Simplicity and scaling benefits of cloud native platforms
Load bal.
Elastically scaled containers
Pooled shared underlying resources,
but decoupled containers
Implicit HA/DR
Deploy by image combining artefacts
and infrastructure
Administer image then redeploy, not
hot fixing.
Orchestration
FrameworkImage
Build
Code
repository Release
Image repository
Authoring
Load bal. Load bal.
Monitoring
Log aggregator
Product specific component
Product specific artefact
Template Image
repository
a
a
a
a
a
a
a
b
b
c
c
c
a
a
27. Cattle
"Cattle not pets: Achieving lightweight integration with IIBâ http://ibm.biz/CattlePetsIIB
Building up the layers in the Docker image
Ubuntu
IIBv10
Fixed config.
Ubuntu Ubuntu
ACEv11
Ubuntu
ACEv11 + MQ
Fixed config.
Ubuntu
ACEv11
Fixed config 1
Ubuntu
ACEv11
bar file A
Fixed config.
Ubuntu
ACEv11
bar file B
Fixed config.
Ubuntu
ACEv11
Fixed config 2
Deployable
units
30. 29
Artefacts
Resources
Security
Operations
Routing
Deployment
Delivery
Traditional infrastructure
(Pets)
Cloud native infrastructure
(Cattle)
Artefacts
Resources
Security
Operations
Routing
Deployment
Delivery
Fine grained
Breaking up into multiple fully
decoupled components
(Architecture & Design)
Cloud native
Standardized administration,
orchestration, monitoring.
(Infrastructure & Technology)
Decentralized
Removing centralized control,
and providing autonomy to teams
(People & Process)
31. Synergy
Architecture
& Design
People &
Process
Infrastructure &
Technology
Maturity
Business
Benefit
Architecture
& Design
People &
Process
Infrastructure &
Technology
Maturity
Business
Benefit
Architecture
& Design
People &
Process
Infrastructure &
Technology
Maturity
Business
Benefit
Architecture
& Design
People &
Process
Infrastructure &
Technology
Maturity
Business
Benefit
32. Common themes across the integration portfolio
enabling microservices principles
âą Lightweight runtimes with a cloud native options
âą Trivial, no/low cost repeatable developer install
âą DevOps toolchain support, scriptable âinfrastructure as codeâ
âą Support for containers, orchestration frameworks
âą Cloud ready, and cloud vendor agnostic
âą Standardized logging to enable cross component correlation
âą New licensing models including flex, hybrid and usage based
âą âDigital connectivityâ â e.g. support for REST, NoSQL, Kafka, SaaS etc.
31
33. Integration architecture â key references
32
List of useful links on agile integration architecture
http://ibm.biz/AgileIntegArchLinks
Original PoV paper on microservices and in integration
Article http://ibm.biz/MicroservicesVsSoa
Webinar http://ibm.biz/MicroservicesVsSoaFullWebinar
Hybrid Integration Reference Architecture
Article http://ibm.biz/HybridIntRefArch
Webinar http://ibm.biz/HybridIntRefArchYouTube
Part 2: Moving to lightweight, agile integration
http://ibm.biz/AgileIntegArchPaper
Part 1: The fate of the ESB
http://ibm.biz/FateOfTheESBPaper