The evolving story for Agile Integration Architecture in 2019

Hursley Summit 2019
Theevolvingstoryfor
AgileIntegrationArchitecture
in2019
Kim Clark
Integration Architect

Organizational decentralization from an integration perspective
Increased team agility as a result of greater
autonomy and clearer ownership
Integration
1) Integration requirements from
application teams are coming in
increasingly rapidly as greater innovation
occurs at the engagement tier
2) The integration team
can’t keep up. It takes
them to long to familiarize
with the domain specific
integration needs.
Furthermore, they are
deploying onto large
shared integration
environments. Centralized integration facility with
plateauing productivity
5) The original integration teams act as consultants to the
application teams to ensure consistent use of the integration
technology. Application developers gradually gain confidence in
the integration capability and become self sufficient
3) Application teams
become disillusioned and
loose faith in the value of
the integration capability
6) Microservices teams begin to
recognize integration runtimes as
more productive in the creation of
some microservice components
4) Integration ownership is progressively
transitioned to the application teams, aided
by the operational consistency resulting
from containerization technology.

Infrastructure & Technology
People & Process
Architecture & Design
Agile Integration
Architecture
Modernizing
integration to enable
business agility
Fine grained
deployment
Improve build
independence and
production velocity
(deployment agility)
Decentralized
Ownership
Accelerate agility and
innovation
(development agility)
Cloud native
infrastructure
Dynamic scalability
and inherent
resilience
(operational agility)

Perspectives on
Agile Integration Architecture
Application
Integration
perspective
API
perspective
M
essaging
perspective
3

Evolution to agile integration (2015-2018 version)
Containerization
Centralized
ESB
Fine-grained
integration
deployment
Application autonomy Polyglot runtimes
Decentralized
integration
ownership
Integration as a
microservice runtime
superseded

Integration
Evolution to agile integration architecture – high level view
5
Centralized
ESB
Fine-grained
integration
deployment
Decentralized
integration
ownership
Containerization Application autonomy
API
API
API
API
API
API GatewayAPI Gateway
API Gateway API Gateway
API APIAPI
Webinar http://ibm.biz/AgileIntegArchPodcasts
eBooklet https://www.ibm.com/cloud/agile-integration-architecture

Integration
Evolution to agile integration architecture – detail view
6
Centralized
ESB
Fine-grained
integration
deployment
Decentralized
integration
ownership
API
Integration
API API
APIAPI
API
API
API
Engagement
applications
Systemsof
record
Eventstream

Runtime upgrade only
Runtime upgrade and
split for basic decoupling
Runtime upgrade and move to
fine grained, cloud-native
App Connect Enterprise
Adoption paths
Server 1
IS
(ACE)
x 70 flows
Integration
Node
IS
(ACE)
x 30 flows
Server 2
IS
(ACE)
x 70 flows
Integration
Node
IS
(ACE)
x 30 flows
Server 1
Integration
Node
Integration
Server
(ACE)
x 100 flows
Server 2
Integration
Node
Integration
Server
(ACE)
x 100 flows
Server 1
Integration
Node
Integration
Server
(IIB/WMB)
x 100 flows
Server 2
Integration
Node
Integration
Server
(IIB/WMB)
x 100 flows
Container
platform
x5
x10
x20
x3
x10
x7
x6
x4
x5 x4
x6
x10
x3
x7
Traditional
basic deployment
Traditional upgraded
with
basic isolation
Cloud native
(full isolation)
Traditional
upgraded
https://developer.ibm.com/integration/blog/2019/04/26/ace-adoption-paths

Availability Zone A
Server PRD1
HA
Manager
What’s the container equivalent of the HA/DR topology you have today?
Container orchestration platform
(multi-zone)
Container
Server PRD2
Availability Zone B
Server DR1
HA
Manager
Server DR2 Infrastructure as code
Replication:
minimum 2
maximum 2
Spread across zones
Balance workload evenly
Traditional
(explicit configuration)
Container platform
(declarative logical configuration)
Re-instate
on failure

Container orchestration platform (multi-zone)
Containers enable fine-grained deployment…and discrete scaling policies
with near-complete abstraction from physical resources
min 3
max 7
min 2
max 2
min 1
max 1
min 1
max 9
min 3
max 3
min 3
max 5

Deployment granularity
First split by business domains and functional areas to ensure high-level autonomy.
Next, consider non-functionals such as a) which need a separate pipeline (for agility),
b) which need independent scalability, c) which have unique resilience requirements
10
All domains
integration artefact integration runtime
Domain 1 Domain 2
Domain 3
before after

integration artefact
integration runtime
before after
Domain 1 Domain 1Shared lifecycle
(e.g. shared data model)
Stabile requirements
(e.g. integrations that haven’t
needed changes in years)
Specific resilience
(e.g. require high replication
factor to achieve availability)
Inter-related availability
(e.g. reliant on the
availability of one another)
Technical dependencies
(e.g. all require local MQ
server due to 2PC)
Elastic scalability
(e.g. occasional high
spikes in usage)
http://ibm.biz/aia-granularity

RS “=n”
Pod
Cont.
SS “=1”
Pod
Cont.
SS “=n”
Pod
Cont.
RS “=n”
Pod
Cont.
PV
Elastic scaling Elastic scaling,
with non-persistent
local MQ
Single instance with
persistent local MQ
Active/active with persistent
independent local MQ
PV
StatefulStateless
Primary container images mapped to core topologies
Container Image
Integration Server
Container ImageContainer Image
Integration Server
MQ Client
Integration Server
Local Queue Manager
https://developer.ibm.com/integration/blog/2018/06/21/ace-need-local-mq-server
RS Replica Set
SS Stateful Set
Cont. Container
PV
Persistent
volume

Benefits of a container based strategy
Containerization is more than just a re-platforming exercise. “lift and shift” will not bring the above benefits.
Requires: Fine-grained deployment, organizational decentralization, pipeline automation, disposable components…
Build Agility
Team
Productivity
Fine-grained
Resilience
Scalability
and
Optimization
Operational
Consistency
Component
Portability

API PortalAPI Portal
system of
record
system of
record
system of
record
Capabilities scale in at differing levels of granularity
application
API Gateway
API Portal
API Management
API PortalAPI Portal
Microservice
application
API Gateway
Microservice
application
integrations
Fine-grained
integration
deployment
Independently
scaled gateway
clusters
Independently
scaled API
management
componentry
API Management
API Analytics
API Analytics

Elastically scaling containers across environments
Production
AppDev
IntDev
SysTest
UAT
Performance

Elastically scaling containers across environments
Production
AppDev
IntDev
SysTest
PoC UAT
Performance

Hardware
Operating system
Language/product
runtime
Topology
Artefacts and fixed
configuration
Environment
config
Build Run
Product
infrastructure
specialist
Base
infrastructure
specialist
Application
Developer
Product
Operations
Base
infrastructure
operations
Traditional
Hardware
Operating system
Language/product
runtime
Topology
Artefacts and fixed
configuration
Environment
config
Build Run
Application
Developer
Operations /
Site Reliability
Engineer (SRE)
Cloud native
Dynamically
provisioned
Platform
managed
Templatized
Automation

Perspectives
Application
Integration
perspective
API
perspective
M
essaging
perspective
18
http://ibm.biz/AgileIntegArchPodcasts

Integration
Evolution to agile integration architecture – detail view
19
Centralized
ESB
Fine-grained
integration
deployment
Decentralized
integration
ownership
API
Integration
API API
APIAPI
API
API
API
Engagement
applications
Systemsof
record
Eventstream

Integration
Provider(s)
Differentiating exposure from implementation
API Gateway
Consumer(s)
Exposure
(consumer focused lifecycle)
Control point: Consistent provision of routing, versioning, traffic
management, security, logging.
Socialization: Enables discovery, documentation, and self-
subscription, analytics.
Implementation
(provider focused lifecycle)
Composition: Implements the custom “integration logic”, including
aggregation from multiple sources, and merging of data.
Adaptation: Understands the connectivity protocols and data
formats, required to manage communication with specific provider
systems.

Decentralized API ownership
Developer Portal
API Manager
API
Implementation
A
API
Implementation
B
API
Implementation
C
A B C
A B C
API gateway API gateway API gateway
on a centralized API management infrastructure
API gateway
21https://developer.ibm.com/apiconnect/2018/12/10/api-management-centralized-or-decentralized

Managing
decentralization
22
API platform/infrastructure team
API
provider
team A
API
provider
team B
API
provider
team C
API
provider
team D
Guild –including members from all teams
API
provider
team A
API
provider
team B
API
provider
team C
API
provider
team D
Secondments to provider teams
API
provider
team A
API
provider
team B
API
provider
team C
API
provider
team D
Communication gap results in divergence

µService
µService
µService µService
µService
API
Application
boundary
µService
Microservice
component
µService
µService
µService
API gateway
API gateway
µService
API gateway
Silo
application
Silo
application
Managed API gateways define and enforce application boundaries
Boundaries make complex environments manageable
µService
µService
µService
API gateway
µService
µService
µService
API gateway
µService
µService
µService
API gateway
23https://developer.ibm.com/apiconnect/2018/10/09/apis-microservices-defining-boundaries
API gateway

Microservice
component
Exposed API
Application
boundary
A service mesh provides capabilities within the application boundary
API gateway
Microservice
component
Microservice
component
Service mesh
proxy
Service mesh
routed requests
Traditional
Application
Microservice
Application
Microservice
Application
API management
gateway
Inter-microservice security
– Authentication,
authorization
Deployment patterns
– A/B, canary
Fault tolerance
– Retries, circuit breaker,
rate-limiting
Visibility
– Logs, Metrics, tracing
Testing
– Fault injection
The service mesh intercepts invocations, enabling policy-based interconnectivity patterns to reduce code, and simplify the
implementation of security, version management, monitoring, diagnostics, testing and more within the microservices application.
https://developer.ibm.com/apiconnect/2018/11/13/service-mesh-vs-api-management

Collaboration between API management and the service mesh
API gateway
v1 v2
MicroserviceApplication
Route 1% of requests from beta and test
consumers only to v2
Beta plan
consumers
Gold plan
consumers
Consumers are known to the API
management gateway by metadata
such as their application key, and
their subscription plan.
Proxies extract the header information
making it available to policies to route
based on API consumer meta-data
Invocation header is extended during
ingress with information including the
consumers key and plan
Traditional
Application
Limit all non-gold plan consumers to 10
requests per second to the back end system
Enable end to end tracing for a particular
consumer key for specific diagnostics
Silver plan
consumers
Test plan
consumers
Inject simulated fault responses for test
consumers only.
Proxies enact policies for routing, security,
tracing, fault injection, rate limiting etc.
How does it work?
What could it do?
https://developer.ibm.com/apiconnect/2018/11/13/service-mesh-vs-api-management

Perspectives
Application
Integration
perspective
API
perspective
M
essaging
perspective
26

Integration modernization
Messaging perspective
27
Traditional application pervasive, self-
managed messaging topologies Self-provisioned, platform-managed, secure, reliably
and transparent communication in a multi-cloud
environments. Delivering an event driven enterprise.

asynchronous(messaging,events,…)
trad. on premisescloud platform
cloud platformcloud platform
private cloudSaaS platform
iPaaS/iSaaS
SaaS platform
cloud platform cloud platform
The asynchronous backplane (messaging, events)
The asynchronous backplane provides reliable message/event storage and a
distribution network that can traverse application and cloud boundaries robustly.
28

Micro services inter-communication
Event
Hub
Publish
Microservice A
Microservice B Microservice C
Subscribe
Event sourcing
Pros: Asynchronously decoupled
Cons: Data duplication, additional persistence
Mitigations: Event stream as data master
Publish
Microservice A
Microservice B Microservice C
API API
Subscribe
Synchronous API calls
Pros: Simple implementation
Cons: Real-time dependencies
Mitigations: Circuit breaker, caching
Neither is perfect. Weigh up pros and cons, and perhaps combine both.

Source / Target
Request / Reply
Stream History
Transient Data
Persistence
Immutable Data
Scalable
Consumption
Targeted
Reliable Delivery
Events
(notifications)
Messaging
(commands)
ü
30
https://developer.ibm.com/messaging/2018/05/18/comparing-messaging-event-streaming-use-cases

trad. on premises
cloud
platform
private cloud
The TCP challenge for large file transfers
As network distances increase, TCP suffers disproportionately with increases in packet loss and latency.
For large file transfers this can result in order of magnitude increases in transfer time over FTP.
31
cloud
platform
cloud
platform
cloud
platform
cloud
platform
cloud
platform
cloud
platform
latency
packet loss
packet loss
packet loss
packet loss
latency
latency
latency
latency
latency

The batch/bulk backplane (file transfer, bulk APIs, …)
The batch/bulk backplane must enable end-to-end, reliable, secure, scalable, interruptible, linear transfer rates across
disparate locations, regardless of geographical or network distance. It should get the data all the way to the last mile using the
most efficient final protocol into the target (e.g. database, Hadoop, block storage, object storage, filestore, …)
on premises cloud platform
cloud platformcloud platform
cloud to cloudcloud to ground
cloud platform
cloud platform
one to many
cloud platform

Moving to agile integration – a real world example
Moving to cloud is a progressive evolution of enterprise architecture, not a big bang
Multiple aspects of integration architecture change along that journey
33
API
API Gateway
API Gateway
API API
API
API
API
API
API
Event streams are made available
such that microservice
applications can build
independent data stores rather
than always having to make real-
time calls over APIs
Applications migrated from on-premise to IaaS cloud, bring their integrations with
them, potentially moving them under the ownership of the application teams
New applications created based
on microservices architecture
Multiple cloud destinations, likely
from multiple cloud vendors
All applications, old and new, expose “managed” APIs such
that they can be consumed by other applications
Existing ESB broken up into
separately deployable integrations
using containerization
Asynchronous ”hub and
spoke” style integrations such
as ”data sync” also split into
separately deployable
containers.
”internal’ APIs always require
managed exposure, but use
integration only where necessary
on-premises
Engagement applications can
consume any API exposed within
the organization, or indeed beyond.
Where a microservices performs
an integration-like job, use a
lightweight integration runtime

34Think 2019 / #7334 / February 10th, 2019 / © 2019 IBM Corporation
Know WHY you want to ‘Be Agile’
Know what outcomes and value
you want Agile to give you.
Remove existing pain points
Embrace the container concept:
Remove dependencies.
Bind in your flows and App
Connect versions.
Same image in Dev, Test, Prod:
Inject configuration
Let your containers run anywhere
Consider your granularity:
it will affect your agility.
Define your deployment and test
boundaries
Find out what changes – and how often!
Delegate NFRs to your Agile Architecture
Use K8s and Service Meshes for endpoint
routing & config
Use K8s for continuous availability
Use centralized logging: The number of
containers will change dynamically.
Key Takeaways

Agile Integration Architecture (AIA)
eBooklet
https://www.ibm.com/cloud/agile-integration-architecture
Webinar series
Other key links on agile integration architecture
http://ibm.biz/AgileIntegArchLinks
Hybrid Integration Reference Architecture
https://ibm.biz/HybridIntRefArch
http://ibm.biz/HybridIntRefArchYouTube
http://ibm.biz/MultiCloudIntegrationArchitectureWebinar
Staying up to date:
https://developer.ibm.com/apiconnect/blog
https://developer.ibm.com/integration/blog
https://developer.ibm.com/messaging/blog
Public material
on integration
modernization

The evolving story for Agile Integration Architecture in 2019

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a The evolving story for Agile Integration Architecture in 2019

Semelhante a The evolving story for Agile Integration Architecture in 2019 (20)

Mais de Kim Clark

Mais de Kim Clark (19)

Último

Último (20)

The evolving story for Agile Integration Architecture in 2019