Canonical data model - how to apply to an existing environment with web services (SOA and REST)

1. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 1)
phil@mp3monster.org
www.mp3monster.org
‘How to implement a canonical data
model in an existing SOA estate’

2. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 2)
phil@mp3monster.org
www.mp3monster.org
Introduction
• The following deck attempts to address the question:
• ‘How to implement a canonical data model in an existing SOA
estate’
• To address this we need to understand a number of
things:
– Assumptions on the current state of affairs
– The value proposition of adopting a canonical model – no
point in an adoption approach that doesn’t deliver value
(with as tangible or intangible benefits)
– The strategies best suited to delivering the goal
– Appreciate the risks we may expose

3. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 3)
phil@mp3monster.org
www.mp3monster.org
Assumptions
• By ‘SOA environment’ we assume to mean capability
centric services primarily built with SOAP/WSDL/XSD and
REST/JSON technologies
• By ‘data model’ we presume to mean data definitions
used in middleware rather than underlying application
and data warehouse/marts
• Assumption that the existing estate doesn’t have an
interface versioning strategy applied across the board
• Services are woven together to deliver larger capabilities
by an ESB
• The approach should be vendor agnostic

4. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 4)
phil@mp3monster.org
www.mp3monster.org
What do we mean by a Canonical Data Model
• The following definition from Forrester 2010 (as part of a
blog on a modelling conference1)
A canonical information model is a model of the semantics and structure
of information that adheres to a set of rules agreed upon within a defined
context for communicating among a set of applications or parties.
• The essence of the various definitions is:
– Internally consistent description of data
– Standard terminology and meaning
– Commonly accepted by all providers & consumers involved in
orchestrating interactions of any form
– Definitions are largely technology agnostic (although typically
not free of the under pinning representation i.e. XML/XSD of
SQL).
1 http://blogs.forrester.com/mike_gilpin/10-03-15-field_first_annual_canonical_model_management_forum

5. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 5)
phil@mp3monster.org
www.mp3monster.org
Value of a Canonical Model
• Semantic Consistency - which allows interactions to have a common
meaning so no problems of your gadget is my widget
– This means mapping data from an event pay load or for WS invocations is easy &
less prone to mapping errors
• Structural Consistency – so the definition of common data items is always
the same
– Eliminates risks of transformation errors
– Potential to reduce transformations in an orchestrated sequence of operations –
meaning greater throughput
• Reduced Design Effort – choose appropriate definitions not create them
– Picking data definitions from a set of models is easier & less error prone than
designing from scratch
• Increases chance of Information Rich integration
– with a predefined data definition increases chances of providing information rich
events as you’re just populating objects
– Information rich events, raises chances of plug and play integration (event types
match, data shared less likely to need changes to get more data)

6. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 6)
phil@mp3monster.org
www.mp3monster.org
Look at a hypothetical integration and how
Canonical Model adoption can change it

7. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 7)
phil@mp3monster.org
www.mp3monster.org
Key
Aggregate
Split
Transform
Store
Tx Endpoint
Endpoint
Pipe (+Filter)
Content Route
De/Normalize
Enrich
Canonical Data
App Data
Icons from Hhope & Woolfe EAIPatterns.com
Organic Growth & Non Canonical Model
• Organic Growth
• No canonical model
• Creating need for
multiple
transforms &
related
operations
• Some operations may
undo previous
operation
• Canonical 
application models
excluded here

8. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 8)
phil@mp3monster.org
www.mp3monster.org
Key
Aggregate
Split
Transform
Store
Tx Endpoint
Endpoint
Pipe (+Filter)
Content Route
De/Normalize
Enrich
Canonical Data
App Data
Icons from Hhope & Woolfe EAIPatterns.com
Same Systems with Canonical Model
(systems not canonical conversant)
• Greatly simplified
as each system is
fronted by a
transform
to/from local
representation to
canonical

9. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 9)
phil@mp3monster.org
www.mp3monster.org
Key
Aggregate
Split
Transform
Store
Tx Endpoint
Endpoint
Pipe (+Filter)
Content Route
De/Normalize
Enrich
Canonical Data
App Data
Icons from Hhope & Woolfe EAIPatterns.com
Same Systems with Canonical Model
(some systems canonical conversant)
• Number of
transformations
reduced
• Middleware
purely becomes a
routing / pub-sub
delivery

10. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 10)
phil@mp3monster.org
www.mp3monster.org
Select or Create own Canonical Model?
• Industry standard models cover wider number of domains, but will
not provide 100% fit all the time
• Creating works …
– In a closed, non SaaS/COTs environment create custom canonical model
by deliver benefits
• closer match to service implementation – performance gain
• Alignment to business language
– Create need to take into account lessons from designing enterprise
application/DB data models
• Select a standard model means
– Leverage accumulated good practise lessons learnt/data needs for
interoperability
– Industry models likely to be 80/20 fit you will need own definitions for
business specific concepts e.g.
• in optical retail need to extend standard definition of Item with definition from
Vision Council of America clinical elements of lens shape & cut
– Model selection(s) need to be done with care
– Make sure model(s) are sufficiently mature

11. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 11)
phil@mp3monster.org
www.mp3monster.org
Technical Strategies / Decisions Needed

12. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 12)
phil@mp3monster.org
www.mp3monster.org
Interface/Payload Versioning
• Interfaces or the payload need to have a versioning
strategy as they will change overtime, can be applied by
– URI – works very well for REST
– XSD schema versioning – common, but a problem for
REST+JSON
• Need to know how many versions to actively support
– common to keep current + 1
– factors to account for rate of change & interface users ability to
accommodate the rate of change
• Determine approach handling
– Common URL + ESB conditional logic
– Separate URLs + ESB logic re-use

13. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 13)
phil@mp3monster.org
www.mp3monster.org
Versioning Existing Interfaces – Some Options
• If existing interface/payload has no version
– ESB can use absence as implicit version 0.
– Requires routing conditional logic
• If interface does exist, then
– If versioning uses same strategy then recommend new interface
URIs
– If different can share URI and use ESB to determine version
– Requires routing conditional logic
• Simply create slightly different URI for replacement
services
– Increased volume of code,
– endpoint user aware of change
(less desirable)
– conditioning is implicit

14. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 14)
phil@mp3monster.org
www.mp3monster.org
Transition States
• There will always be a period of transitionary state when
adopting major change such as canonical model
– Therefore when passing on or starting a sequence of event(s)
what do I assume about down stream capability?
• This can be addressed by one of several strategies:
– Late binding using UDDI or equivalent and discover service and
version of interface available – great if overhead is not a
problem
– Assume latest version (predicated on ability to transform to
previous version) & programme of work provides proxy to
legacy interface which transforms down
– Software controlled switching of output (not desirable as
embeds knowledge of consumers into a service)

15. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 15)
phil@mp3monster.org
www.mp3monster.org
REST+JSON Question
• Canonical models in support of the middleware are
typical XSD based today - good for SOAP WSDL services
but REST + JSON more challenging as no schema needed
• However organisations starting to offer JSON models e.g.
part of OASIS, OAGIS
• Could use REST + XML (more like RPC than proper REST)
• Could publish JSON mapped representation (tooling
available) with description via JSON Schema (IETF draft)
– Safest when offering special custom service
– Still delivers benefit for internal services
• Remember R (in REST) is for Resource and ideally you
want resources to be consistent in definition

16. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 16)
phil@mp3monster.org
www.mp3monster.org
Challenges of Abstraction vs Endpoint Needs
• Current interfaces maybe geared towards supporting
specific platforms e.g.
– Phone, thick client, IoT (Internet of Things i.e. agent
devices such as smart meters)
– E.g strip generic message to only necessary elements as
device can only handle small payloads
• Strategy for this is to add layer between core
canonical & ESB an endpoint aware transformation
– Means core routing / business aspects of ESB not impacted
– so changing routing etc not impacted
– Clients not needing adaptation can talk directly to
canonical layer

17. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 17)
phil@mp3monster.org
www.mp3monster.org
Governance
• Engaging with all these previous considerations will
need to be factored into any Governance processes..
– Design Time
• Assurance that the correct approaches identified are being
adopted
• Adoption is for the right reasons
– Execution Time
• Ability to ascertain the adoption, efficacy etc
• We started out with the
declaration that we’re working in a
SOA context, which should mean
– SOA Governance is in place and can
support these Governance goals Open Group
http://www.opengroup.org/soa/source-book/gov/sgvm_artifacts.jpg

18. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 18)
phil@mp3monster.org
www.mp3monster.org
Understanding Why & How To Adopt
Canonical Model we can look at execution

19. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 19)
phil@mp3monster.org
www.mp3monster.org
Implementation Strategy
• Define architectural strategies (i.e. engage with
previous rational and challenges)
• Need to ensure ground work is in place to enable
correct development, could be delivered by
– Reference implementation
– Documentation set with policy & practise
– Very detailed requirements (including test definitions)
– Support through architect involved in pair programming
– workshops

20. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 20)
phil@mp3monster.org
www.mp3monster.org
Implementation Strategy (1)
• Start small & grow as …
– Knowledge & understanding develops
– Principles, ideas and approach are refined
– Help manage risk & impact
– Can make ensure initial work is ‘referencable’
• Assess & Measure
– Helps build cost/benefit -- ROI insights (both hard and soft
factors)
– Informs planning & estimation downstream
– Ensure implementation quality & sustainability

21. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 21)
phil@mp3monster.org
www.mp3monster.org
TOGAF View of realising Canonical Model
Defined objective for how and
why a Canonical Model be
adopted (assuming other
principles etc already set)
Establish business direction of
travel so we can identify
suitable model(s), opportunities
for a pilot
Determine key business
data structures
Build the Tech Ref
Model & Stds
Information Base
Look at opportunities
for piloting canonical
adoption
As not greenfield
transition strategy is
needed
Hands on support
– key to identify
lessons and
approaches to
accelerate & ease
adoption
Apply refinements
to pilot. Depending
upon scope plan
next cycle
Set direction of travel,
scope for change

22. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 22)
phil@mp3monster.org
www.mp3monster.org
Activities from a Execution Sequence
Perspective
Identify Canonical model(s)
& Strategies
•Which model(s) to use
•Approaches/impact to handling
transition
Develop Model Knowledge
•Interpretation/Language
•Versioning
•Ensure guidance & supporting
information is ready
Determine non Critical
integration programme
•Scope cover various patterns of
use and impact
•Agree benchmarks to establish
value
•Develop detailed implementation
plan
Start Development
•Ensure testing of existing
interfaces are in place so can
assure of no impact
•Develop initial interfaces inc
interface & e2e tests
Regression Testing
•Ensure that different message
types & versions exercised
•Check for changes in type within
end to end execution
•Deployment strategy included
Performance & Other
PreProd Tests
•Canonical models can be heavier –
therefore ensure performance is
considered
•Assess Value of approach
Assuming Success…
•Expand adoption
•KT to wider team etc
•Programme of full adoption
Establish Monitoring
•Need to determine when legacy
interfaces stop being used
•Retire interfaces at appropriate
time
Iterate
development
process

23. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 23)
phil@mp3monster.org
www.mp3monster.org
Reminder & Questions

24. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 24)
phil@mp3monster.org
www.mp3monster.org
Remember!
• This has been done before – so make sure you’re
considering best practise recommendations
(particularly from preferred vendors)

25. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 25)
phil@mp3monster.org
www.mp3monster.org
Questions

26. ‘How to implement a canonical data model in an existing SOA
estate’
19/05/2014 (slide 26)
phil@mp3monster.org
www.mp3monster.org
Thankyou