Introduction to RAG (Retrieval Augmented Generation) and its application
SOA Pattern: Data Model Transformation
1. Last Updated: Jan. 2014
Associate Technical Lead – Technical
Sales
Dakshitha Ratnayake
SOA Pattern: Data
Model Transformation
2. 2
About the Presenter
๏ Dakshitha is an Associate Technical Lead in the
Solutions Architecture/Technical Sales team.
๏ She has hands on experience in Java/J2EE technologies
and has worked on designing and implementing
software solutions in the fields of health-care
information systems and content management systems
for telecommunications providers prior to her
employment at WSO2. Prior to joining the Solutions
Architecture/Technical Sales team she worked with the
WSO2 Developer Studio team.
3. 3
About WSO2
๏ Global enterprise, founded in
2005 by acknowledged leaders
in XML, web services
technologies, standards and open
source
๏ Provides only open source
platform-as-a-service for private,
public and hybrid cloud
deployments
๏ All WSO2 products are 100% open
source and released under the
Apache License Version 2.0.
๏ Is an Active Member of OASIS,
Cloud Security Alliance, OSGi
Alliance, AMQP Working Group,
OpenID Foundation and W3C.
๏ Driven by Innovation
๏ Launched first open source API
Management solution in 2012
๏ Launched App Factory in 2Q 2013
๏ Launched Enterprise Store and
first open source Mobile solution
in 4Q 2013
6. The Integration Problem
๏ As organizations move forward with their SOA
implementation, they find themselves grappling with
significant problems of data integration among a wide
variety of data sources.
๏ The mismatch between the data representation that one
application provides and the data representation that
another application expects forms the core of this long-lived
problem.
๏ This data integration problem is particularly troublesome for
SOA implementations because it limits the loose coupling
between service providers and consumers.
8. A Narrowed Down Problem
๏ How can services interact with programs that
communicate with different data formats?
๏ Services may use incompatible schemas to represent the same data, hindering
service interaction and composition.
๏ Even when organizations use metadata standards such as XML to format
messages, the schema of one Service might not match the schema of another,
and fields with the same meaning might use differing tags.
9. The Straight-Forward
Approach
๏ When an application or service must transform data
from one format to another in order to achieve some
desired data integration goal, the most common and
straightforward approach is to embed data
transformation logic directly within the application.
10. Disadvantages of Using
Embedded Transformation
๏ Other applications or Services have no practical way of
reusing the embedded transformations.
๏ It's impossible to offload the CPU overhead that the
embedded data transformations cause.
๏ The transformation logic is tightly coupled to the business
logic within the application, reducing the flexibility and
reusability of the applications.
๏ The transformation logic is handled programmatically vs.
declaratively through metadata, further impeding the
flexibility of the application.
11. A Better Approach
๏ An alternative to embedding transformation logic within an
application is to use an external transformation service to
handle the task.
๏ A transformation service, loosely-coupled in a SOA
implementation, accepts incoming data in one format, and
outputs data with the same semantic meaning in another
format.
๏ The use of data transformation services improves loose
coupling by separating the concerns of service providers and
consumers.
12. Solution for ‘same format different
schema’ problem
๏ A data transformation technology can be incorporated
to convert data between disparate schema structures.
๏ Mapping logic needs to be developed and deployed so that data
compliant to one data model can be dynamically converted to comply
to a different data model.
13. Solution Pattern: Data Model
Transformation
๏ The requirements statement for Data Model
Transformation is:
How can services interoperate when using different
data models for the same type of data?
14. Reference Architecture: Standard
Integration with XSLT transformation
Service 2Service 2
Service 1Service 1
Data Model
Transformation
Data Model
Transformation
Schema1.xsdSchema1.xsd
Schema2.xsdSchema2.xsd
Schema1.xsdSchema1.xsd
Schema2.xsdSchema2.xsd
Transform.xslTransform.xsl
15. WSO2 Solution Architecture:
Standard Integration
Service 2Service 2
Service 1Service 1 Schema1.xsdSchema1.xsd
Schema2.xsdSchema2.xsd
Schema1.xsdSchema1.xsd
Schema2.xsdSchema2.xsd
Transform.xslTransform.xsl
16. WSO2 Offerings
The WSO2 ESB supports transformations using the following
transformation mediators:
WSO2 Data Services Server supports transformation using XSLT
Transformation, where the user can define the transformation in
XSLT.
17. Entity Aggregation
๏ There are essentially three reasons for the aggregation of
entities in different systems:
๏ Single View of Entity—Having a single view of all common
entities spread across enterprise systems.
๏ Horizontal Partitions—There are situations where entity
information is distributed across many different services,
based on factors such as geographical boundaries.
๏ Cross-Join scenarios—These situations require cross-join
among entities present in different services.
18. Reference Architecture: Entity
Aggregation
ERP ServiceERP Service
Data Model
Transformation
(Entity Aggregation)
Data Model
Transformation
(Entity Aggregation)
Customer
ERPCustID
LName
Fname
OrgID
Customer
ERPCustID
LName
Fname
OrgID
Transform.xslTransform.xsl
CustomerAPICustomerAPI
CRM ServiceCRM Service
Customer
CRMCustID
LName
Fname
Address
Tel
OrgID
Customer
CRMCustID
LName
Fname
Address
Tel
OrgID
Customer
ERPCustID
LName
Fname
OrgID
Address
Tel
Customer
ERPCustID
LName
Fname
OrgID
Address
Tel
19. WSO2 Solution Architecture: Entity
Aggregation
ERP ServiceERP Service
Customer
ERPCustID
LName
Fname
OrgID
Customer
ERPCustID
LName
Fname
OrgID
Transform.xslTransform.xsl
CustomerAPICustomerAPI
CRM ServiceCRM Service
Customer
CRMCustID
LName
Fname
Address
Tel
OrgID
Customer
CRMCustID
LName
Fname
Address
Tel
OrgID
Customer
ERPCustID
LName
Fname
OrgID
Address
Tel
Customer
ERPCustID
LName
Fname
OrgID
Address
Tel
20. Standards Compliance
๏ Adherence to standards insures, to some degree, the
loose coupling and composition of services and data
exchange mechanisms with a guarantee that the data
exchanged with an individual service will be universally
comprehensible.
๏ E.g. – HL7, other domain-specific formats
As businesses continued to expand over the years, their demand for more IT systems increased. As mergers and acquisitions became the standard, there was an even greater demand for new systems. As a result, today's enterprises are composed of a plethora of IT systems that are independent of each other. The result is hundreds, if not thousands, of application islands.
When services and service consumer programs interact, data is transmitted (usually in the form of messages) organized according to some structure and a set of rules. This structure and the associated rules constitute a formal representation (or model) of the data.
When different services are designed with different data models representing the same type of data, then they will have a problem sharing this data because the data models are simply incompatible. To address this problem, a technique called data model transformation is applied whereby data model mapping logic is developed so that data exchanged by such services is dynamically converted at runtime from compliance with one data model to another. So successful has this technique been that a corresponding Data Model Transformation pattern was developed.
An XSLT style sheet containing data model mapping logic (2) is added as a form of intermediary processing that is executed at runtime. With each transmission, the data model of the service 1 is converted from schema1 to the data model compliant with the schema2 used by service 2. This runtime transformation logic can reside with either service architecture or as part of a separate middleware platform.
through XSLT transformation on the XML payload and message stream, XQuery transformations, message enrichment and message content replacement
There are essentially three reasons for the aggregation of entities in different systems:
Single View of Entity—Having a single view of all common entities spread across enterprise systems. E.g a unified view of a Customer entity, named Customer Information Repository (CIR), is required to build useful solutions in many enterprises, especially in the banking sector.
Horizontal Partitions—There are situations where entity information is distributed across many different services, based on factors such as geographical boundaries.
Cross-Join scenarios—These situations require cross-join among entities present in different services.
Goal of an aggregation service-
Acts as a unified source of entities.
Provides a holistic view of an entity.
Provides a holistic view of the entity model—entities and their relationships with other entities
Provides location transparency—consumers of the Entity Aggregation layer do not need to know who owns the information.
Enforces business rules that determine the segments of entities retrieved in a given context.
Determines the system of record for each data element constituting an entity.
Enriches the combined data model across systems—the whole being better than the sum of its parts.
Business concepts such as Customer and Employee were duplicatedFor example, in many businesses employees must make separate phone calls to HR, Payroll, and Benefits to have their addresses updated in the different systems.
An Entity Aggregation solution can be designed to not only act as a single point in accessing information that exists in multiple systems, but it also provides a holistic view of an entity and the entity model.
Not only do disparate services have different views of an entity, they may also have their own schematic representation of their views. In order to achieve a single unified view of an entity, it is imperative for the EA service to harmonize the schematic differences between different services. To do this, the EA service should first know how each service represents an entity. Additionally, the EA service should define a holistic view of the entity by logically consolidating the various views. Equally important is for the EA service to discover a way to represent the transformation between a service's view of an entity and the holistic view. The following figure shows the customer information represented in more than one service.