Presentation at 8th Workshop on Distributed Software Development, Software Ecosystems and Systems-of-Systems - WDES 2014 Website: http://wdes2014.icmc.usp.br/Index_en Proceedings (per paper): http://www.lbd.dcc.ufmg.br/bdbcomp/servlet/Evento?id=787 Proceedings (volume): http://www.ic.ufal.br/evento/cbsoft2014/anais/wdes_v1_p.pdf

1. Towards the Evaluation of Systems of
Systems Architecture
Daniel S. Santos
Brauner R. N. Oliveira
Milena Guessi
Flavio Oquendo
Marcio Delamaro
Elisa Yumi Nakagawa

2. Schedule
• Introduction
• Systems of Systems
• Architecture Evaluation Methods
• Evaluation of SoS Software Architectures
• Evaluation Methods
• Quality attributes
• Challenges
• Conclusion
Towards the Evaluation of Systems of Systems Architecutre
CBSoft 2014 – Workshop em Desenvolvimento Distribuído de
Software, Ecossistemas de Software e Sistemas de Sistemas – 2

3. Introduction
Systems of Systems
"A set or arrangement of independent, useful systems integrated into
larger systems that deliver unique capabilities " (DoD, 2008).
Challenges:
• Constituents systems are often developed and
maintained by different organizations.
• These organizations can have their own stakeholders,
development teams, and processes.
• Evaluation of SoS software architectures could ensure
that quality attributes are satisfied
Towards the Evaluation of Systems of Systems Architecutre
CBSoft 2014 – Workshop em Desenvolvimento Distribuído de
Software, Ecossistemas de Software e Sistemas de Sistemas – 3

4. Introduction
Architecture Evaluation
• Can be used to compare and identify strengths and weaknesses of
different architectural alternatives
• Guide the maintenance or indicate new opportunities for
enhancing software architectures
• It is essential for ensuring that software architectures meet
desired quality attributes
Towards the Evaluation of Systems of Systems Architecutre
CBSoft 2014 – Workshop em Desenvolvimento Distribuído de
Software, Ecossistemas de Software e Sistemas de Sistemas – 4

5. Introduction
• Goals: To present the state of the art on SoS software
architecture evaluation as well as new research perspectives.
• Based on results of a Systematic Literature Review (SLR)1.
• Overall, 16 primary studies were included in this SLR.
1 Available at http://goo.gl/PU12iQ (last accessed on 07/13/2014)
Towards the Evaluation of Systems of Systems Architecutre
CBSoft 2014 – Workshop em Desenvolvimento Distribuído de
Software, Ecossistemas de Software e Sistemas de Sistemas – 5

6. Primary studies selected
Towards the Evaluation of Systems of Systems Architecutre
CBSoft 2014 – Workshop em Desenvolvimento Distribuído de
Software, Ecossistemas de Software e Sistemas de Sistemas – 6

7. Evaluation of SoS Software Architecture
Evaluation is performed in:
• Design phase
• Comparing and identifying strengths and weaknesses of
different architectural alternatives.
• Maintenance phase
• Intending to analyze the flexibility and ability to evolution
Towards the Evaluation of Systems of Systems Architecutre
CBSoft 2014 – Workshop em Desenvolvimento Distribuído de
Software, Ecossistemas de Software e Sistemas de Sistemas – 7

8. Evaluation Methods
• No convergence in using a specific type of
evaluation method.
• Architecture Trade-off Analysis Method
(ATAM) is the most popular.
Towards the Evaluation of Systems of Systems Architecutre
CBSoft 2014 – Workshop em Desenvolvimento Distribuído de
Software, Ecossistemas de Software e Sistemas de Sistemas – 8
Frequency of use of methods and techniques

9. Maturity of Evaluation Methods
Towards the Evaluation of Systems of Systems Architecutre
CBSoft 2014 – Workshop em Desenvolvimento Distribuído de
Software, Ecossistemas de Software e Sistemas de Sistemas – 9
31%
13%
31%
19%
6%
Demonstration
Expert Opinion
Case Study
Application in
Industry
No Assessment
Maturity of the approaches

10. Quality Attributes
Evaluation methods can either focus on single or several quality attributes.
• Several Quality Attributes
• Scenario-based methods
• No focus on specific quality attributes
• Few tangible quality attributes
• simulation-based and mathematical modeling methods
• Reliability, performance, complexity, and flexibility.
• None of the studies discuss the use of quality models during
architectural evaluation.
Towards the Evaluation of Systems of Systems Architecutre
CBSoft 2014 – Workshop em Desenvolvimento Distribuído de
Software, Ecossistemas de Software e Sistemas de Sistemas – 10

11. Quality Attributes
Recommendations
• Evaluation methods for SoS should take into account several quality attributes.
• Evaluation methods should be able of measuring and classifying quality
attributes.
• Quality models would be relevant for evaluating SoS architectures
• Provide standardization for quality attributes
• Guide the identification and prioritization of quality attributes
Towards the Evaluation of Systems of Systems Architecutre
CBSoft 2014 – Workshop em Desenvolvimento Distribuído de
Software, Ecossistemas de Software e Sistemas de Sistemas – 11

12. Conclusion
• There is still no consensus on what exactly should be considered during
evaluation.
• There is still several difficulties and challenges
• Appropriate and scalable evaluation approaches still need to be
developed.
Towards the Evaluation of Systems of Systems Architecutre
CBSoft 2014 – Workshop em Desenvolvimento Distribuído de
Software, Ecossistemas de Software e Sistemas de Sistemas – 12

13. Towards the Evaluation of Systems of
Systems Architecture
Daniel S. Santos
Brauner R. N. Oliveira
Milena Guessi
Flavio Oquendo
Marcio Delamaro
Elisa Yumi Nakagawa