This presentation is part of the course "184.742 Advanced Services Engineering" at The Vienna University of Technology, in Winter Semester 2012. Check the course at: http://www.infosys.tuwien.ac.at/teaching/courses/ase/
This PowerPoint helps students to consider the concept of infinity.
TUW - 184.742 Data marketplaces: models and concepts
1. Advanced Services Engineering,
WS 2012, Lecture 6
Data marketplaces: models and concepts
Hong-Linh Truong
Distributed Systems Group,
Vienna University of Technology
truong@dsg.tuwien.ac.at
http://www.infosys.tuwien.ac.at/staff/truong
ASE WS 2012 1
3. Recall – data service units in
clouds/internet
data data
data
Data service unit Data service unit Data service unit
data
People Things
Internet/Cloud
ASE WS 2012 3
4. Recall – data as a service
Data-as-a-Service – service models
Data publish/subcription Database-as-a-Service
middleware as a service (Structured/non-structured
querying systems)
Sensor-as-a-Service Storage-as-a-Service
(Basic storage functions)
deploy
Private/Public/Hybrid/Community Clouds
ASE WS 2012 4
5. Data marketplaces
More than just DaaS
DaaS focuses on data provisioning features
Data marketplaces
Multiple data providers and consumers
Multiple DaaS
Complex interactions among DaaS, data providers
and consumers
Complex billing and pricing models
Market dynamics
ASE WS 2012 5
7. Some important issues
DAAS DESCRIPTION MODEL
DATA AGREEMENT EXCHANGE
DATA CONTRACT
ASE WS 2012 7
8. Description Model for DaaS (1)
State of the art:
Providers have their own way to describe DaaS,
mainly in HTML
Existing service description techniques are not
adequate in supporting description for DaaS
Problems
Service and data discovery cannot be done
automatically
On-demand data integration, service integration, and
query optimization cannot be supported well.
Service/data information and DaaS engineering
cannot be tied.
ASE WS 2012 8
9. Description Model for DaaS (2)
Which levels must be covered?
Here
Data resource
Data
items
Consumer
Data Data
Data items items
assets
Consumer
Data resource Data resource
Data resource Data resource
DaaS
ASE WS 2012 9
10. Description Model for DaaS – types
of information
Which types of information must be covered?
Quality of Ownership
data Price
License ....
Service
interface
Service
license
Quality of
service ....
ASE WS 2012 10
11. DEMOS – a description model for
Data-as-a-Service
Quang Hieu Vu, Tran Vu Pham, Hong
Linh Truong,, Schahram Dustdar,
Rasool Asal: DEMODS: A Description
Model for Data-as-a-Service. AINA
2012: 605-612
See prototype:
http://www.infosys.tuwien.ac.at/
prototype/SOD1/demods/
ASE WS 2012 11
15. Discussion time
WHICH TYPES OF DAAS INFORMATION
ARE DYNAMIC? AND THEIR IMPACT ON
DESCRIPTION MODELS?
ASE WS 2012 15
16. Exchange data agreement (1)
Consumer DaaS provider Data
provider
DaaS
Consumer
DaaS
DaaS
Sensor
How they interact w.r.t. data concerns?
How their data agreements look like?
ASE WS 2012 16
17. Exchange data agreement (2)
Lack of models and protocols for data
agreement in data marketplaces
Constraints for data usage are not clear
Inadequate data/service description → hindering data
selection and integration
Existing techniques are not adequate for
dynamic data agreement exchange in data
marketplaces
Need generic exchange models suitable for different
ways of data provisioning in data marketplaces
ASE WS 2012 17
18. Data Agreement Exchange as a
Service (DAES)
Metamodel for data agreement exchange
Techniques for enriching and associating data
assets with agreement terms
Interaction models for data agreement exchange
Hong Linh Truong, Schahram Dustdar, Joachim Götze, Tino Fleuren, Paul Müller, Salah-Eddine Tbahriti, Michael Mrissa,
Chirine Ghedira: Exchanging Data Agreements in the DaaS Model. APSCC 2011: 153-160
ASE WS 2012 18
19. Metamodel for data agreements
Different
category of
agreements
Licensing,
privacy, quality
of data
Extensions
Languages
Different types
of agreements
Different
specifications
ASE WS 2012 19
20. Associating data with data
agreements
Solutions
(a) directly inserting agreements into data assets
(b) providing two-step access to agreements and data
assets
(c) linking data agreements to the description of DaaS
(d) linking data agreements to the message sent by
DaaS
ASE WS 2012 20
22. DAES – conceptual architecture
Jersey, JAX-RS Restful WS Weblogic
Using URIs to identify agreements
ASE WS 2012 22
23. DAES – managed information
Specific applications: agreement creation, agreement validation,
agreement compatibility analysis, agreement management
Implementation: Jersey, JAX-RS Restful WS Weblogic
ASE WS 2012 23
24. Illustrating examples – insert
agreement into data asset
A pay-per-use consumer uses dataAPI of DaaS
search for data
The consumer pays the use APIs
Each call can return different types of data
Example with
People Search in
Infochimps
But a strong consequence
for data service engineering
techniques: dealing with
elastic requirements!
ASE WS 2012 24
25. Illustrating examples – link
agreements to geospatial data
Domain-specific DaaS: different agreements for different data requests
Vector data of geographic features via Web-Feature-Service (WFS)
Terrain elevation data via Web-Coverage Services (WCS)
ASE WS 2012 25
26. Illustrating examples – link
agreements to geospatial data
Consumers can interpret and
reason if the data can be
used for specific purposes
ASE WS 2012 26
28. Illustrative examples – develop an
app for policy compliance (2)
Configuration
Results
ASE WS 2012 28
29. Discussion time
HOW NEAR-REALTIME DATA IMPACTS ON
DATA AGREEMENT EXCHANGE?
ASE WS 2012 29
30. Data contract
How to specific data contract?
Data resource
Data
items
Consumer
Data Data
Data items items
assets
Consumer
Data resource Data resource
Data resource Data resource
DaaS
ASE WS 2012 30
31. Data contracts
Give a clear information about data usage
Have a remedy against the consumer where the
circumstances are such that the acts complained
of do not
Limit the liability of data providers in case of
failure of the provided data;
Specify information on data delivery,
acceptance, and payment
ASE WS 2012 31
32. Data contracts
Well-researched contracts for services but not
for DaaS and data marketplaces
But service APIs != data APIs =! data assets
Several open questions
Right to use data? Quality of data in the data
agreement? Search based on data contract? Etc.
➔ Require extensible models
➔ Capture contractual terms for data contracts
➔ Support (semi-)automatic data service/data selection
techniques.
Hong-Linh Truong, Marco Comerio, Flavio De Paoli, G.R. Gangadharan, Schahram Dustdar, "Data Contracts for
Cloud-based Data Marketplaces ", International Journal of Computational Science and Engineering, 2012 Vol.7, No.4,
pp.280 - 295
ASE WS 2012 32
33. Study of main data contract terms
Data rights
Derivation, Collection, Reproduction, Attribution
Quality of Data (QoD)
Not mentioned, Not clear how to establish QoD metrics
Regulatory Compliance
Sarbanes-Oxley, EU data protection directive, etc.
Pricing model
Different models, pricing for data APIs and for data assets
Control and Relationship
Evolution terms, support terms, limitation of liability, etc
Most information is in human-readable form
ASE WS 2012 33
35. Developing data contracts in cloud-
based data marketplaces
Follow community-based approach for data
contract
Propose generic structures to represent data
contract terms and abstract data contracts
Develop frameworks for data contract applications
Incorporate data contracts into data-as-a-service
description
Develop data contract applications
ASE WS 2012 35
36. Community view on data contract
development
Community users can develop:
Term categories, term names, values, and units
Rules for data contracts
Common contract and contract fragments
Community users =!
novice users
ASE WS 2012 36
37. Representing data contract terms
Contract term: (termName,termValue)
Term name: common terms or user-specific terms
Term value: a single value, a set, or a range
ASE WS 2012 37
38. Structuring abstract data contracts
Concrete data contracts generates
can be in RDF, XML or
JSON
Use Identifiers and
Tags for identifying
and searches
ASE WS 2012 38
39. Development of contract
applications
Main applications:
Data contract compatibility evaluation, data contract
composition
Some common steps
Extract DCTermType in TermCategoryType
Extact comprable terms from all contracts,
- e.g., dataRight: Derivation, Composition and Reproduction
Use evaluation rules associated with DCTermType
from from rule repositories
Execute rules by passing comparable terms to rules
Aggregate results
ASE WS 2012 39
40. Prototype
RDF for representing term categories, term
names, term values, units
Allegro Graph for storing contract knowledge
ASE WS 2012 40
41. Illustrating examples
A large sustainability monitoring data platform
shows how green buildings are
Real-time total and per capita of CO2 emission
of monitored building
Open government data about CO2 per capita at
national level
We created contracts from
Open Data Commons Attribution License
Open Government License
ASE WS 2012 41
43. Step 2: provide OpenBuildingCO2
OpenBuildingCO2 by OpenGov for
modifying quality of government data
data and data right
Data contract for green building data
ASE WS 2012 43
45. Discussion time
CAN WE AUTOMATICALLY GENERATE
DATA CONTRACTS FOR NEAR-REALTIME
DATA?
ASE WS 2012 45
46. Exercises
Read mentioned papers
Examine existing data marketplaces and write
DEMODS-based specification for some of them
Develop some specific data contracts for open
government data
Work on some algorithms for checking data
contract compatiblity
ASE WS 2012 46
47. Thanks for
your attention
Hong-Linh Truong
Distributed Systems Group
Vienna University of Technology
truong@dsg.tuwien.ac.at
http://www.infosys.tuwien.ac.at/staff/truong
ASE WS 2012 47