In Quest for Requirements Engineering Oracles: Dependent Variables and Measurements for a (good) RE
1. Technische Universität Mßnchen
In Quest for Requirements Engineering Oracles
DependentVariables and Measurements for (good) RE
Joint work with
Jakob Mund
Henning Femmer
AntonioVetrò
Daniel MĂŠndez FernĂĄndez
!
Technische Universität Mßnchen
Germany
!
EASE 2014
London, UK
13.05.2014
@mendezfe
2. A regular case study in requirements engineeringâŚ
Goal: Test the sensitivity of a new RE method in company context
Âť Introduce a new RE method in project environment
Âť Conduct action research workshops following the new method
Âť Rate the method outcome according to assessment criteria deďŹned using GQM
! [Mendez Fernandez et al.A Case Study on the Application of an â¨
Artefact-Based Requirements Engineering Approach]
3. A regular case study in requirements engineeringâŚ
What conclusions can I draw beyond the chosen context?
!
Âť How representative are the variables / how important are they to other contexts?
Âť How can the variables be efďŹciently measured?
Âť What relations exist between the variables?
4. Dependent variables in requirements engineering
Why?
Status quo in requirements engineering research
⢠Development of methods/tools for problems too often not properly understood
Âť Control of validity / threat mitigation?
!
Research objectives
⢠Get a better understanding on RE phenomena, their dependencies and their
measurability
!
Our motivation
Understand RE phenomena and research implications on evidence-based RE research
⢠DifďŹculties arising from cause-effect relationships
⢠Measurability and actionability
⢠Implications on threat to validity (e.g. possibilities of generalisation)
5. Study design
Research questions
!RQ 1. Which RE-related phenomena exist, where in the project ecosystem â¨
do they manifest themselves, and how do they relate to each other?
RQ 2. Are the phenomena measurable?
RQ 3. Are the phenomena actionable?
6. Study design
Research questions
!
Requirements Engineering survey
NaPiRE www.re-survey.org
Globally distributed, replicated survey â¨
on status quo and problems in RE
RQ 1. Which RE-related phenomena exist, where in the project ecosystem â¨
do they manifest themselves, and how do they relate to each other?
RQ 2. Are the phenomena measurable?
RQ 3. Are the phenomena actionable?
7. Study design
Research questions
!
Requirements Engineering survey
NaPiRE www.re-survey.org
Globally distributed, replicated survey â¨
on status quo and problems in RE
RQ 1. Which RE-related phenomena exist, where in the project ecosystem â¨
do they manifest themselves, and how do they relate to each other?
Researchers
Levels of dependent variables (âDimensionsâ)
[Gorschek et al. Requirements Eng.: In Search of the dependent variables]
RQ 1
RQ 2. Are the phenomena measurable?
RQ 3. Are the phenomena actionable?
RE Phenomena inferred from open â¨
questions on problems, â¨
their causes and their effects
8. Study design
Research questions
!
Requirements Engineering survey
NaPiRE www.re-survey.org
Globally distributed, replicated survey â¨
on status quo and problems in RE
RQ 1. Which RE-related phenomena exist, where in the project ecosystem â¨
do they manifest themselves, and how do they relate to each other?
Researchers
Levels of dependent variables (âDimensionsâ)
[Gorschek et al. Requirements Eng.: In Search of the dependent variables]
RQ 1
RQ 2. Are the phenomena measurable?
RQ 3. Are the phenomena actionable?
Voting of
⢠Measurability (artefacts? activities? both? none?)
⢠Actionability
RQ 2 & 3
RE Phenomena inferred from open â¨
questions on problems, â¨
their causes and their effects
9. Results
RQ 1: Dependent variables
!
!
!
!
⢠93 variables with 167 dependencies
Âť Full result set: http://goo.gl/W3A1tu
Company
Project
Engineering
2
2
22
2
2
2
3
2
2
4
2
3
8
3
2
2
4
2
2
2
2
2
2
2
2
2
3
2
3
3
Requirements
Engineering
M36
Name
Actionable (A)
Measurable (on aRtefacts, on aCtivities,
on aRtefacts & aCtivities)
M03
R
M08
R
R03
R
M09
R
RP06
R
M32
C
RP02
R
M29
C
M30
R&C
M05
C
M15
M20
M13
A
R11
R18
R&C
M24
R
RP10
R&C A
M17
C
RP03
C A
M02
R A
RP08
R A
RP11
A
RP07
R
RP04
C
R01
RP05
R A
M28
RP01
R
[Results with minimum occurrence in elements > 1]
14. SW Project
Engineering
âIncreased effort in testingâ
âŚ
Requirements Engineering
Results (Excerpt)
RQ 1: Dependent variables
âIncomplete requirementsâ
âImplicit requirements not!
made explicitâ âChange requestsâ
âEffort and time overrunâ
âMoving targetsâ
15. SW Project
Engineering
âIncreased effort in testingâ
âŚ
Requirements Engineering
Results (Excerpt)
RQ 1: Dependent variables
âIncomplete requirementsâ
âImplicit requirements not!
made explicitâ
âFailed acceptanceâ
âŚ
âChange requestsâ
âEffort and time overrunâ
âMoving targetsâ
16. Company âCustomer (dis-)satisfactionâ
SW Project
Engineering
âIncreased effort in testingâ
âŚ
Requirements Engineering
Results (Excerpt)
RQ 1: Dependent variables
âIncomplete requirementsâ
âImplicit requirements not!
made explicitâ
âFailed acceptanceâ
âŚ
âChange requestsâ
âEffort and time overrunâ
âMoving targetsâ
17. âIncomplete requirementsâ
âFailed acceptanceâ
âIncreased effort in testingâ
âEffort and time overrunâ
âImplicit requirements not!
made explicitâ
âCustomer (dis-)satisfactionâ
âChange requestsâ
Requirements Engineering
Engineering
SW Project
Company
âŚ
âŚ
âMoving targetsâ
Results (Excerpt)
RQ 2: Measurability
⢠High degree of measurability for variables in RE dimension (59% of variables for RE)
⢠Unmeasurable variables comprise social phenomena (e.g.,âWeak access customer needsâ)
⢠Most measurable variables are measurable on basis of artefacts â¨
(e.g.,âUnderspeciďŹed requirementsâ)
⢠Decreasing understanding of variables and measurability the âfarther awayâ from RE
18. âIncomplete requirementsâ
âFailed acceptanceâ
âIncreased effort in testingâ
âEffort and time overrunâ
âImplicit requirements not!
made explicitâ
âCustomer (dis-)satisfactionâ
âChange requestsâ
Requirements Engineering
Engineering
SW Project
Company
âŚ
âŚ
âMoving targetsâ
Critical reďŹection
The good: Measurability in RE context
19. âIncomplete requirementsâ
âFailed acceptanceâ
âIncreased effort in testingâ
âEffort and time overrunâ
âImplicit requirements not!
made explicitâ
âCustomer (dis-)satisfactionâ
âChange requestsâ
Requirements Engineering
Engineering
SW Project
Company
âŚ
âŚ
âMoving targetsâ
Critical reďŹection
The good: Measurability in RE context
⢠High degree of measurability for RE-related variables
⢠Measurability (to a large extent) on basis of artefacts
Âť Support of independent and comparable measurements
Âť Already a better understanding of RE study context and its characteristics
20. âIncomplete requirementsâ
âFailed acceptanceâ
âIncreased effort in testingâ
âEffort and time overrunâ
âImplicit requirements not!
made explicitâ
âCustomer (dis-)satisfactionâ
âChange requestsâ
Requirements Engineering
Engineering
SW Project
Company
âŚ
âŚ
âMoving targetsâ
Critical reďŹection
The bad: Limitations in RE context
21. âIncomplete requirementsâ
âFailed acceptanceâ
âIncreased effort in testingâ
âEffort and time overrunâ
âImplicit requirements not!
made explicitâ
âCustomer (dis-)satisfactionâ
âChange requestsâ
Requirements Engineering
Engineering
SW Project
Company
âŚ
âŚ
âMoving targetsâ
Critical reďŹection
The bad: Limitations in RE context
?
⢠Incomplete dependencies
Âť Investigation was performed âinside-outâ from RE and needs to be extended
22. âIncomplete requirementsâ
âFailed acceptanceâ
âIncreased effort in testingâ
âEffort and time overrunâ
âImplicit requirements not!
made explicitâ
âCustomer (dis-)satisfactionâ
âChange requestsâ
Requirements Engineering
Engineering
SW Project
Company
âŚ
âŚ
âMoving targetsâ
Critical reďŹection
The bad: Limitations in RE context
⢠Complex (yet incomplete) system of variables
Âť Too many (especially unknown) variables intersect with each other
?
?
⢠Incomplete dependencies
Âť Investigation was performed âinside-outâ from RE and needs to be extended
23. âIncomplete requirementsâ
âFailed acceptanceâ
âIncreased effort in testingâ
âEffort and time overrunâ
âImplicit requirements not!
made explicitâ
âCustomer (dis-)satisfactionâ
âChange requestsâ
Requirements Engineering
Engineering
SW Project
Company
âŚ
âŚ
âMoving targetsâ
Critical reďŹection
The bad: Limitations in RE context
⢠Complex (yet incomplete) system of variables
Âť Too many (especially unknown) variables intersect with each other
⢠Even known variables strongly depend on subjectivity
Âť Requirements engineering is also about beliefs, desires, experiences, expectations and politics
?
?
⢠Incomplete dependencies
Âť Investigation was performed âinside-outâ from RE and needs to be extended
24. âIncomplete requirementsâ
âFailed acceptanceâ
âIncreased effort in testingâ
âEffort and time overrunâ
âImplicit requirements not!
made explicitâ
âCustomer (dis-)satisfactionâ
âChange requestsâ
Requirements Engineering
Engineering
SW Project
Company
âŚ
âŚ
âMoving targetsâ
Critical reďŹection
The ugly: No RE oracle in sight!
No âgeneralisabilityâ â¨
(by no means)
⢠Complex (yet incomplete) system of variables
⢠Even known variables strongly depend on subjectivity
⢠Incomplete dependencies
25. Research implications (Excerpt)
General implications on evidence-based RE research
⢠No generalisation possible, but ďŹrst set of variables and ideas for measurability
Âť Aim for universal language rather than for universal truth
⢠High degree of subjectivity
Âť Qualitative studies (e.g. expert judgment)
26. Research implications (Excerpt)
General implications on evidence-based RE research
⢠No generalisation possible, but ďŹrst set of variables and ideas for measurability
Âť Aim for universal language rather than for universal truth
⢠High degree of subjectivity
Âť Qualitative studies (e.g. expert judgment)
Implications on research on RE methodologies
⢠Inference of ďŹrst improvement goals
Âť CalibrateâŚ
â Study designs according to variables and their measurement
â Variables if aiming for tackling âcommonly acceptedâ problems
27. Threats to Validity
⢠Variables are incomplete (needless to say)
⢠ClassiďŹcationâŚ.
⢠to some extent with a high disagreement
⢠not the only possible way (and itâs positivistic)
!
!
⢠However: It is a way to learn more and we are
conďŹdent to further continue it!
28. !
Full data disclosed to:
http://goo.gl/W3A1tu
!
⢠raw data
⢠full graphs
⢠R scripts
⢠âŚ
@mendezfe
⢠Next replication of NaPiRE ongoing (www.re-survey.org)
Âť Gather more RE phenomena
⢠from NaPiRE
⢠from other sources (e.g. via longitudinal studies)
!
!
!
Future work
daniel.Mendez@tum.de
Letâs build a big picture of RE variables together!
29. Backup
* Cohenâs Kappa values for classiďŹcation
A phenomenon is measurable, if its understanding is â¨
sufďŹciently mature such that
(i)  an existing or anticipated measure, i.e., objective map- â¨
ping to mathematical objects
(ii) can efďŹciently (e.g., in justiďŹable time) and effectively â¨
(i.e., preserving empirical observations) capture the â¨
phenomenon
(iii)  under practical conditions and when applied on study â¨
objects which can be expected to be present in a soft- â¨
ware project ecosystem.