1. Analysing the concept of quality in
model-driven engineering literature:
a systematic review
Fáber D. Giraldo,Sergio España and Oscar Pastor
RCIS 2014
Marrakesh,Morocco,June 2014

2. Hi, I am Fáber
Giraldo

3. Agenda
 Introduction & Motivation
 Systematic review design
 Results achieved
 Discussion

8. Framework by Lindland,Sindre and Sølvberg [1994]
LANGUAGE
AUDIENCE
INTERPRETATION
DOMAIN MODEL
PRAGMATICS
SYNTAX
SEMANTICS
completeness=77% completeness=85%

9. UC 7
PUBLISHING HOUSE
RETURNS SIGNED REPORT
DELIVERY NOTE
UC 13
SALESMAN ISSUES
PUBLISHING HOUSE
INVOICES
SALESMAN
PUBLISHING
HOUSE
UC 13. BASIC FLOW OF ACTIONS:
1. REVIEW DELIVERY NOTES
2. RETRIEVE PUB.HOUSE DATA
3. CALCULATE TOTAL AMOUNT
4. SEND INVOICE TO PUB.HOUSE
PHO 7
PUBLISHING HOUSE
RETURNS SIGNED
REPORT DELIVERY NOTE
SALESMAN
PHO 13
SALESMAN ISSUES
PUBLISHING HOUSE
INVOICES
SALESMAN
SALESMAN
PUBLISHING
HOUSE
PUBLISHING
HOUSE
INVOICE
SIGNED D.N.

10. UC 7
PUBLISHING HOUSE
RETURNS SIGNED REPORT
DELIVERY NOTE
UC 13
SALESMAN ISSUES
PUBLISHING HOUSE
INVOICES
SALESMAN
PUBLISHING
HOUSE
UC 13. BASIC FLOW OF ACTIONS:
1. REVIEW DELIVERY NOTES
2. RETRIEVE PUB.HOUSE DATA
3. CALCULATE TOTAL AMOUNT
4. SEND INVOICE TO PUB.HOUSE
PHO 7
PUBLISHING HOUSE
RETURNS SIGNED
REPORT DELIVERY NOTE
SALESMAN
PHO 13
SALESMAN ISSUES
PUBLISHING HOUSE
INVOICES
SALESMAN
SALESMAN
PUBLISHING
HOUSE
PUBLISHING
HOUSE
INVOICE
SIGNED D.N.
completeness=76% completeness=81%
N=35

14. MOF
ATL
METAMODEL
CA2UML
TRANSFORMATION
COMMUNICATION
ANALYSIS
METAMODEL
UML2 CLASS
DIAGRAM
METAMODEL
REQUIREMENTS
MODEL
UML
CLASS
DIAGRAM
Conforms to
Conforms to Conforms to
Based on Conforms toConforms to Based on
input
executed
output
Conforms to

15. Introduction & Motivation (I)
 Many works on model quality
– Moody and ISO 9126
– SEQUAL (Krosgtie, Lindland, Solverg)
– Good practices in modelling (Mohagheghi)
– UML defects and guidelines
 Most proposals are very theoretical (scarce validation).
 Lack of conceptual consensus (what is model quality?).
 It is unclear whether model-driven engineering principles
are considered.
– Complementary modellinglanguages
– Model transformation
– Code generation

16. Introduction & Motivation (II)
 Problematic phenomena:
– Existing MDE standards do not address quality
– No standard quality framework for MDE
 Research goal:
– Review literature on model / modelling language quality
– Structure proposals
– Find out their applicability to MDE
– Discover trends

17. Agenda
 Introduction & Motivation
 Systematic review design
 Results achieved
 Discussion

18. Systematic review design (I)
 Kitchenham’s guidelines for systematic review
 Research questions:
– (RQ1) What does quality mean in the context of MDE
literature?
– (RQ2) What does it mean to say that an artifact
conforms to the principles of MDE?
– (RQ3) Are current quality-related methods within MDE
able to deal with a set of modeling languages?

19. Systematic review design (II)
 Search
– Scientific databases (ACM Digital Lib., IEEExplore, Springer, Scopus, etc.)
– Indexing services (Google Scholar, DBLP )
– Conference Proceedings (CAISE, ER, RCIS, RE, etc.)
– Industrial repositories (OMG)
The process of reading and selecting was performed by a PhD student
under the supervision of two senior researchers

20. Results achieved (I)
 Selection
– Inclusion criteria
– Exclusion criteria
First results Depuration

21. Systematic review design (III)
 Data extraction
– Classification schemas to analyse retrieved studies include:
1. Kinds of artefacts whose quality is measured (e.g. models, modelling
languages, sets of modelling languages)
2. Kinds of artefacts proposed to measure quality (e.g. methods and tools)
3. Degree of validation and operationalisation of 2.

22. Agenda
 Introduction & Motivation
 Systematic review design
 Results achieved
 Discussion

23. Results achieved (II)

24. Results achieved (III)
 We identified 11 trends on the studies
– Grouping of studies depending on the theory on which they build.
0 0 0 0 0 0 0 0 0 0 0 0
3
4
5
2
11
0 0 0 0 0
1
0
1 1
0
1
0 0
1 1 1
0 0 0 0 0 0
1
3
0
2
3
4 4
2
1
0 00 0 0 0 0 0 0 0 0 0 0 0
1
0 0 0 00 0 0 0 0 0 0 0 0 0
1
2
0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0
1
3
1
0 00 0 0 0 0 0 0 0
1 1 1
0 0 0 0
1 1
0 0 0 0 0 0
1 1
3
4
3 3
1 1
0 0 00 0 0 0 0 0 0 0 0 0
1 1
2
0
1
0 00 0 0 0 0 0 0 0
1
2
1 1
0 0 0 0 00 0 0 0 0 0
1
0 0 0 0 0 0 0 0 0 0
0
1
2
3
4
5
6
1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
Numberofidentifiedstudies
Year
Studies published by trends and years
MDWE SEQUAL 6C UML guid. MoSMe Tranform.
Empir. UML Underst. UML OPM2SysML Harel&Rumpe Structural

25. Results achieved (III)
 We studied the influence on some authors on others
– We applied a sampling technique to identify influential studies

26. Agenda
 Introduction & Motivation
 Systematic review design
 Results achieved
 Discussion

27. Discussion (II)
 (RQ1) What does quality mean in the context of MDE
literature?
– There are many studies on model/modelling language quality
• Most focus on UML
• There is no agreement on the definitionof quality(ill-defined)
– MDE communityhas paid little attention to the topic so far
– The definitions of quality in the trends do not converge
– Many quality evaluation methods and metrics are ad-hoc and
specific for assessing a method created by the same authors
• To prove the benefits of their method (comparative experiments)
• To provide a suitable qualityevaluation method
– Few quality evaluation methods are supported by tools
• Arendt and Taentze complainabout the difficulty

28. Discussion (II)
 (RQ2) What does it mean to say that an artifact conforms
to the principles of MDE?
– This remains unclear.
– The retrieved definitions about quality in models are a strong
basis to start the discussion
– However, most of the MDE core features are not considered in
depth
• suitability of languages to engineering goal
• suitability of a set of languages used in combination
• conformity to MDE principles
• managementof abstraction levels
• model granularity

29. Discussion (II)
 (RQ3) Are current quality-related methods within the
MDE able to deal with a set of modeling languages?
– Complex information systems require multiple viewpoints
supported by conceptual models (different modelling
languages)
– Practically all studies deal with a single language
– Those who consider a set of languages do not provide a
systematic, repeatable and generalisable method to evaluate
any set of languages
–  The evaluation of a set of modelling languages used in
combination remains an open research question.

30. Discussion (IV)
 Interesting future works
– Operationalise model quality proposals using technological
frameworks such as EMF, MetaEdit...
– Compare model quality proposals when applied to specific
MDE contexts
• Laboratory research (controlled experiment)
• Field research (action-research, case study)
– Design a method for evaluating quality in MDE contexts
(building on top of existing proposals)
• Conceptual framework
• Methodological guidance
• Tool support

31. Ask me the
difficult
questions
fdgiraldo@pros.upv.es

32. Analysing the concept of quality in model-driven
engineering literature: a systematic review
Fáber D. Giraldo, Sergio España and Oscar Pastor
RCIS 2014
Marrakesh, Morocco,June 2014