The document summarizes research into understanding software quality requirements throughout a project's lifecycle by analyzing data from eight open source Gnome projects. It aims to test Lehman's law that quality appears to decline over time, examine if high-level quality requirements are treated equally across projects, and determine if the methodology is viable. The researchers used word lists and taxonomies to identify quality-related concepts in project artifacts and analyzed occurrences over time to address the research questions. The results provided limited support for Lehman's law and showed quality requirement treatment varies between projects. Threats to the approach's validity and future work were also discussed.

1. On the perception of software quality
requirements during the project lifecycle
Neil A. Ernst, John Mylopoulos
{nernst,jm}@cs.toronto.edu
University of Toronto
July 1, 2010
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 1 / 16

2. Motivation
On the perception of software quality requirements during the project
lifecycle
Throughout the lifecycle
Understand software quality requirements using an empirical,
data mining approach
Quality requirements as universal requirements
Dataset: 8 Gnome OSS projects
Data / code available at
http://github.com/neilernst/gnome-quality-mining
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 2 / 16

3. Specific research questions:
1 Test Lehman’s 7th Law1 : system quality will appear to be
declining unless rigorously maintained and adapted to
environmental changes
2 Are high-level quality requirements (e.g., efficiency) of equal
concern in different projects?
3 Is this a viable mechanism for understanding the treatment of
quality requirements?
1
Lehman, M.M., et al.: Metrics and laws of software evolution – the nineties view. In: International Software Metrics
Symposium pp. 20–32. Albuquerque, NM (1997)
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 3 / 16

4. Specific research questions:
1 Test Lehman’s 7th Law1 : system quality will appear to be
declining unless rigorously maintained and adapted to
environmental changes
2 Are high-level quality requirements (e.g., efficiency) of equal
concern in different projects?
3 Is this a viable mechanism for understanding the treatment of
quality requirements?
1
Lehman, M.M., et al.: Metrics and laws of software evolution – the nineties view. In: International Software Metrics
Symposium pp. 20–32. Albuquerque, NM (1997)
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 3 / 16

5. Specific research questions:
1 Test Lehman’s 7th Law1 : system quality will appear to be
declining unless rigorously maintained and adapted to
environmental changes
2 Are high-level quality requirements (e.g., efficiency) of equal
concern in different projects?
3 Is this a viable mechanism for understanding the treatment of
quality requirements?
1
Lehman, M.M., et al.: Metrics and laws of software evolution – the nineties view. In: International Software Metrics
Symposium pp. 20–32. Albuquerque, NM (1997)
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 3 / 16

6. Methodology
ISO9126,
Gnome projects:
Wordnet, Boehm,
commits, bugs,
McCall, KDE
mail
Create end
Generate Query
signifiers
corpora corpora
(WN,ext)
start Plot
Precision/recall Error outcomes
on random analysis and calculate
events stats
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 4 / 16

7. Datasets
Product Language Size (ksloc) Age (years) Type
Evolution C 313 10.75 Mail
Nautilus C 108 10.75 File mgr
Metacity C 66 7.5 Window mgr
Ekiga C++ 54 7 VOIP
Totem C 49 6.33 Media
Deskbar Python 21 3.2 Widgets/UI
Evince C 66 9.75 Doc viewer
Empathy C 55 1.5 IM
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 5 / 16

8. Signifiers and quality requirements
Cross-project comparisons using a standardized taxonomy – ISO
9126.
Find quality concepts in project artifacts as signifier occurrences.
Corpora: emails, bug reports, commit messages.
Database of 900k ‘messages’ (total, all projects).
This menu is
barely usable!
email
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 6 / 16

9. Signifiers
Detect the appearance of quality concepts in the messages.
Created signifier lists, a representative signifier thesaurus.
Two lists: one based on Wordnet and ISO9126, the other
extended with domain knowledge (ext). E.g.,
Quality Signifiers
Maintainability testability changeability analyzability stability maintainability
maintain maintainable modularity modifiability understandabil-
ity
Functionality security compliance accuracy interoperability suitability func-
tional practicality functionality
Portability conformance adaptability replaceability installability portable
movableness movability portability
Efficiency “resource behaviour” “time behaviour” efficient efficiency
Usability operability understandability learnability useable usable ser-
viceable usefulness utility useableness usableness serviceable-
ness serviceability usability
Reliability “fault tolerance” recoverability maturity reliable dependable
responsibleness responsibility reliableness reliability dependable-
ness dependability
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 7 / 16

10. Observations
1 Search for events
2 Normalize for message volume
3 Group by week
Event Quality
...By upgrading to a newer version of GNOME you None
could receive bug fixes and new functionality.
There should be a feature added that allows you to Functionality
keep the current functionality for those on work-
stations (automatic hot-sync) and then another
option that allows you to manually initiate .
Steps to reproduce the crash: 1. Can’t reproduce Reliability,
with accuracy. Seemingly random. .... Functionality
These calls are just a waste of time for client Efficiency
and server, and the Nautilus online storage view
is slowed down by this wastefulness.
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 8 / 16

11. Error analysis
1 Pick random messages; manually classify
2 Compare manual classification to machine classification
3 Precision: 80%, Recall: 51% (extended), 6% Wordnet
Signified quality Precision ext Precision WN
Usability 0.47 0.22
Portability 0.11 0.20
Maintainability 0.22 0.31
Reliability 0.15 0.19
Functionality 0.14 0.18
Efficiency 0.16 0.07
Mean 0.21 0.20
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 9 / 16

12. Error analysis
1 Pick random messages; manually classify
2 Compare manual classification to machine classification
3 Precision: 80%, Recall: 51% (extended), 6% Wordnet
Signified quality Precision ext Precision WN
Usability 0.47 0.22
Portability 0.11 0.20
Maintainability 0.22 0.31
Reliability 0.15 0.19
Functionality 0.14 0.18
Efficiency 0.16 0.07
Mean 0.21 0.20
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 9 / 16

13. Observations
RQ1: Lehman’s law
Is there a relation between events and project lifespan?
Looked at three possible explanations:
Overall lifespan
Project release windows
Key external events
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 10 / 16

14. Occurrences (normalized)
0 100 200 300 400
1998
Gnome 1.0
Ernst, Mylopoulos (University of Toronto)
Gnome 1.2
2000
Gnome 1.4
2002
Gnome 2.0
Gnome 2.2
Gnome 2.4
Year
Gnome 2.6
2004
Gnome 2.8
Perception of quality req.
Gnome 2.10 Gnome 2.10
Gnome 2.12
Evolution -- Usability (R2 = 18%)
2006
Gnome 2.16
Gnome 2.20
2008
Gnome 2.24
July 1, 2010
11 / 16

15. Results
Correlation coefficient very low in both cases - no relationship
Analyzing peaks seems useful - external events are driving
frequencies
“... I know this was discussed a few weeks ago ... could it be
implemented as an advanced option that has to be turned on
and is off by default?”
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 12 / 16

16. Observations (2)
RQ2: Are qualities treated the same in different projects?
Examine normalized occurrences per week for extended signifiers
Quality Project Occurrences
Efficiency Evolution 0.012
Nautilus 0.026
Usability Evolution 0.192
Nautilus 0.285
Portability Evolution 0.010
Nautilus 0.011
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 13 / 16

17. Threats to validity
1 How valid are word lists?
2 Domain-dependent quality models
3 Other sources of project discussions (IRC/F2F/Skype)
4 Open source projects may not generalize
5 No sense of ‘attitudes’ to requirements
6 Statistics
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 14 / 16

18. Threats to validity
1 How valid are word lists?
2 Domain-dependent quality models
3 Other sources of project discussions (IRC/F2F/Skype)
4 Open source projects may not generalize
5 No sense of ‘attitudes’ to requirements
6 Statistics
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 14 / 16

19. Threats to validity
1 How valid are word lists?
2 Domain-dependent quality models
3 Other sources of project discussions (IRC/F2F/Skype)
4 Open source projects may not generalize
5 No sense of ‘attitudes’ to requirements
6 Statistics
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 14 / 16

20. Threats to validity
1 How valid are word lists?
2 Domain-dependent quality models
3 Other sources of project discussions (IRC/F2F/Skype)
4 Open source projects may not generalize
5 No sense of ‘attitudes’ to requirements
6 Statistics
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 14 / 16

21. Threats to validity
1 How valid are word lists?
2 Domain-dependent quality models
3 Other sources of project discussions (IRC/F2F/Skype)
4 Open source projects may not generalize
5 No sense of ‘attitudes’ to requirements
6 Statistics
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 14 / 16

22. Threats to validity
1 How valid are word lists?
2 Domain-dependent quality models
3 Other sources of project discussions (IRC/F2F/Skype)
4 Open source projects may not generalize
5 No sense of ‘attitudes’ to requirements
6 Statistics
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 14 / 16

23. Threats to validity
1 How valid are word lists?
2 Domain-dependent quality models
3 Other sources of project discussions (IRC/F2F/Skype)
4 Open source projects may not generalize
5 No sense of ‘attitudes’ to requirements
6 Statistics
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 14 / 16

24. Future work: agenda
RQ3: Usefulness: Technique worked, but:
1 Leverage other machine learning techniques like multi-label
learning
2 Understand inter-project variations and usefulness of global
taxonomies of quality
3 Use techniques to understand requirements evolution in software
projects
4 Tie experimental data to ethnographic data2
2
Aranda, J. and Venolia, G.: The secret life of bugs: Going past the errors and omissions in software repositories. In
International Conference on Software Engineering, pp 298-308 (2009).
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 15 / 16

25. Happy Canada Day!
Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 16 / 16