The document discusses an empirical study that examines the impact of professional status and expertise on comprehension of UML class diagrams. It aims to determine whether years of experience or professional status is a better indicator of expertise by comparing the performance of students, junior professionals, and senior professionals on UML class diagram comprehension tasks. The study analyzed the results to answer four research questions related to the effect of status, expertise, their relationship, and question precision.

1. Professional status
vs. Expertise
Z´phyrin Soh et al.
e
Problem and
Motivations Professional Status and Expertise for UML
Problem
Motivations Class Diagram Comprehension:
Related Work
Expertise Studies An Empirical Study
UML Class Diagram
Comprehension
Empirical Study
Study Design
Z´phyrin Soh, Zohreh Sharafi, Bertrand Van den Plas,
e
Results
RQ1: Status
Gerardo Cepeda Porras, Yann-Ga¨l Gu´h´neuc and
e e e
RQ2: Expertise
RQ3: Status vs. Expertise
Giuliano Antoniol
RQ4: Question Precision
Conclusion and Department of Computer and Software Engineering
Future Work ´
Ecole Polytechnique de Montr´al, Qu´bec, Canada
e e
Conclusion
Threats to Validity and
Future Work
June 13, 2012
Pattern Trace Identification, Detection, and Enhancement in Java
SOftware Cost-effective Change and Evolution Research Lab

2. Professional status
vs. Expertise Outline
Z´phyrin Soh et al.
e
Problem and Motivations
Problem and
Motivations
Problem
Problem
Motivations
Motivations
Related Work Related Work
Expertise Studies
UML Class Diagram
Expertise Studies
Comprehension
UML Class Diagram Comprehension
Empirical Study
Study Design Empirical Study
Results Study Design
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
Results
RQ4: Question Precision RQ1: Status
Conclusion and
Future Work
RQ2: Expertise
Conclusion RQ3: Status vs. Expertise
Threats to Validity and
Future Work
RQ4: Question Precision
Conclusion and Future Work
Conclusion
Threats to Validity and Future Work
2 / 24

3. Professional status
vs. Expertise Problem and Motivations
Z´phyrin Soh et al.
e Problem (1/1)
Problem and
Motivations What is experience?
Problem
Motivations To manage subject/programmer experience:
Related Work
Expertise Studies Years and education as main criteria [1]
UML Class Diagram
Comprehension
Authors sometime combine many criteria
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
[1] J. Feigenspan et al., Measuring Programming Experience, ICPC 2012, pp.
3 / 24 73-82.

4. Professional status
vs. Expertise Problem and Motivations
Z´phyrin Soh et al.
e Motivations (1/1)
Problem and
Motivations
Problem
Motivations
Related Work Motivations
Expertise Studies
UML Class Diagram Consider two following cases:
Comprehension
Empirical Study
A student who used UML for 4 years during her study
Study Design A professional with 3 years of experience with UML
Results
RQ1: Status
Who is the best at understanding of UML class
RQ2: Expertise
RQ3: Status vs. Expertise
diagrams?
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
4 / 24

5. Professional status
vs. Expertise Problem and Motivations
Z´phyrin Soh et al.
e Motivations (1/1)
Problem and
Motivations
Problem
Motivations
Related Work Motivations
Expertise Studies
UML Class Diagram Consider two following cases:
Comprehension
Empirical Study
A student who used UML for 4 years during her study
Study Design A professional with 3 years of experience with UML
Results
RQ1: Status
Who is the best at understanding of UML class
RQ2: Expertise
RQ3: Status vs. Expertise
diagrams?
RQ4: Question Precision
Project managers when recruiting a new software
Conclusion and
Future Work designer by prioritized the important “factor”
Conclusion
Threats to Validity and
Future Work
Future designers to know “where” to acquire the
competitive skills by considering the important “factor”
4 / 24

6. Professional status
vs. Expertise Related Work
Z´phyrin Soh et al.
e Expertise Studies (1/3)
Problem and
Motivations Previous work on expertise
Problem
Motivations Novices spend less time than experts analysts [2]
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
[2] K. D. Schenk, N. P. Vitalari, and K. S. Davis, Differences between novice
and expert systems analysts: what do we know and what do we do?, Journal
of Management Information System, vol. 15, pp. 9-50, June 1998
5 / 24

7. Professional status
vs. Expertise Related Work
Z´phyrin Soh et al.
e Expertise Studies (1/3)
Problem and
Motivations Previous work on expertise
Problem
Motivations Novices spend less time than experts analysts [2]
Related Work
Expertise Studies Graduate students are faster than junior ones and
UML Class Diagram
Comprehension intermediate professionals [3]
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
[3] E. Arisholm and D. I. K. Sjøberg, Evaluating the effect of a delegated
versus centralized control style on the maintainability of object-oriented
software,IEEE Transactions on Software Engineering, vol. 30, no. 8, pp.
5 / 24 521-534, aug. 2004

8. Professional status
vs. Expertise Related Work
Z´phyrin Soh et al.
e Expertise Studies (1/3)
Problem and
Motivations Previous work on expertise
Problem
Motivations Novices spend less time than experts analysts [2]
Related Work
Expertise Studies Graduate students are faster than junior ones and
UML Class Diagram
Comprehension intermediate professionals [3]
Empirical Study
Study Design
Experts are better for abstract questions and novices are
Results better for concrete questions [4]
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
[4] B. Adelson, When novices surpass experts: The difficulty of a task may
increase with expertise,Journal of Experimental Psychology: Learning,
Memory, and Cognition, vol. 10, pp. 483-495, Jul. 1984
5 / 24

9. Professional status
vs. Expertise Related Work
Z´phyrin Soh et al.
e Expertise Studies (1/3)
Problem and
Motivations Previous work on expertise
Problem
Motivations Novices spend less time than experts analysts [2]
Related Work
Expertise Studies Graduate students are faster than junior ones and
UML Class Diagram
Comprehension intermediate professionals [3]
Empirical Study
Study Design
Experts are better for abstract questions and novices are
Results better for concrete questions [4]
RQ1: Status
RQ2: Expertise Experts and novices have different program model for
RQ3: Status vs. Expertise
RQ4: Question Precision
documentation task, no difference for reuse task [5]
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
[5] J.-M. Burkhardt, F. D´tienne, and S. Wiedenbeck, Object-oriented
e
program comprehension: Effect of expertise, task and phase, Empirical
Software Engineering, vol. 7, no. 2, pp. 115-156, 2002
5 / 24

10. Professional status
vs. Expertise Related Work
Z´phyrin Soh et al.
e Expertise Studies (2/3)
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies Comparison
UML Class Diagram
Comprehension
To compare our work with previous work, we consider:
Empirical Study
Study Design
Results
RQ1: Status
⇒ Object
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision ⇒ Kind of task/question
Conclusion and
Future Work
Conclusion ⇒ Subject categorisation criterion
Threats to Validity and
Future Work
6 / 24

11. Professional status
vs. Expertise Related Work
Z´phyrin Soh et al.
e Expertise Studies (2/3)
Problem and
Motivations Comparison
Problem
Motivations
Related Work Ref.
Expertise Studies
UML Class Diagram years of experience
Comprehension requirements
[2] textual description analysis
rating scale of
Empirical Study supervisors
Study Design
students and
Results [3] Java program change task professionals
RQ1: Status
RQ2: Expertise
undergraduate
program + abstract +
RQ3: Status vs. Expertise
RQ4: Question Precision
[4] flowcharts concrete question
students
fellow teachers
Conclusion and
Future Work students and ex-
documentation
Conclusion [5] database program
and reuse
perts (nomination
Threats to Validity and
Future Work
by colleagues, ...)
Legend: Same to our work | Different to our work
[2] Schenk et al. (1998)
[3] Arisholm and Sjøberg (2004)
[4] Adelson (1984)
6 / 24
[5] Burkhardt et al. (2002)

12. Professional status
vs. Expertise Related Work
Z´phyrin Soh et al.
e Expertise Studies (3/3)
Problem and
Motivations Limitations
Problem
Motivations Previous work:
Related Work
Expertise Studies
Did not precisely distinguish years of experience and
UML Class Diagram
Comprehension
professionalism:
Empirical Study
Inexperienced students as novices [5]
Study Design Senior professionals with less years of programming
Results experience than graduate students [3]
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
[5] Burkhardt et al. (2002)
7 / 24 [3] Arisholm and Sjøberg (2004)

13. Professional status
vs. Expertise Related Work
Z´phyrin Soh et al.
e Expertise Studies (3/3)
Problem and
Motivations Limitations
Problem
Motivations Previous work:
Related Work
Expertise Studies
Did not precisely distinguish years of experience and
UML Class Diagram
Comprehension
professionalism:
Empirical Study
Inexperienced students as novices [5]
Study Design Senior professionals with less years of programming
Results experience than graduate students [3]
RQ1: Status
RQ2: Expertise We distinguish the years of experience from
RQ3: Status vs. Expertise
RQ4: Question Precision
professionalism
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
[5] Burkhardt et al. (2002)
7 / 24 [3] Arisholm and Sjøberg (2004)

14. Professional status
vs. Expertise Related Work
Z´phyrin Soh et al.
e Expertise Studies (3/3)
Problem and
Motivations Limitations
Problem
Motivations Previous work:
Related Work
Expertise Studies
Did not precisely distinguish years of experience and
UML Class Diagram
Comprehension
professionalism:
Empirical Study
Inexperienced students as novices [5]
Study Design Senior professionals with less years of programming
Results experience than graduate students [3]
RQ1: Status
RQ2: Expertise We distinguish the years of experience from
RQ3: Status vs. Expertise
RQ4: Question Precision
professionalism
Conclusion and
Future Work Studied the source code or textual descriptions of
Conclusion
Threats to Validity and
Future Work
requirements
[5] Burkhardt et al. (2002)
7 / 24 [3] Arisholm and Sjøberg (2004)

15. Professional status
vs. Expertise Related Work
Z´phyrin Soh et al.
e Expertise Studies (3/3)
Problem and
Motivations Limitations
Problem
Motivations Previous work:
Related Work
Expertise Studies
Did not precisely distinguish years of experience and
UML Class Diagram
Comprehension
professionalism:
Empirical Study
Inexperienced students as novices [5]
Study Design Senior professionals with less years of programming
Results experience than graduate students [3]
RQ1: Status
RQ2: Expertise We distinguish the years of experience from
RQ3: Status vs. Expertise
RQ4: Question Precision
professionalism
Conclusion and
Future Work Studied the source code or textual descriptions of
Conclusion
Threats to Validity and
Future Work
requirements
We use the UML class diagram
[5] Burkhardt et al. (2002)
7 / 24 [3] Arisholm and Sjøberg (2004)

16. Professional status
vs. Expertise Related Work
Z´phyrin Soh et al.
e UML Class Diagram Comprehension (1/2)
Problem and
Motivations UML and eye-tracking
Problem
Motivations Stereotype, color, and layout facilitate class diagram
Related Work exploration and comprehension [6]
Expertise Studies
UML Class Diagram
Comprehension Multi-cluster (by requirement) and three-cluster (by
Empirical Study stereotype) layout positively affect the comprehension of
Study Design
class diagrams [7]
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
[6] S. Yusuf, H. Kagdi, and J. I. Maletic, Assessing the comprehension of UML
diagrams via eye tracking, ICPC’07
[7] B. Sharif and J. I. Maletic, An empirical study on the comprehension of
stereotyped UML class diagram layouts, ICPC’09
8 / 24

17. Professional status
vs. Expertise Related Work
Z´phyrin Soh et al.
e UML Class Diagram Comprehension (1/2)
Problem and
Motivations UML and eye-tracking
Problem
Motivations Stereotype, color, and layout facilitate class diagram
Related Work exploration and comprehension [6]
Expertise Studies
UML Class Diagram
Comprehension Multi-cluster (by requirement) and three-cluster (by
Empirical Study stereotype) layout positively affect the comprehension of
Study Design
class diagrams [7]
Results
RQ1: Status
RQ2: Expertise
Canonical representation of the Visitor pattern in class
RQ3: Status vs. Expertise diagram reduce the effort of maintenance task [8]
RQ4: Question Precision
Conclusion and The representations of design patterns affect the
Future Work
Conclusion
identification of their participants and their roles [9]
Threats to Validity and
Future Work
[8] S. Jeanmart, Y.-G. Gu´h´neuc, H. Sahraoui, and N. Habra, Impact of the
e e
visitor pattern on program comprehension and maintenance, ESEM’09, Oct
2009, pp. 69-78
[9] G. Cepeda Porras and Y.-G. Gu´h´neuc, An empirical study on the
e e
efficiency of different design pattern representations in UML class diagrams,
8 / 24 Empirical Software Engineering, vol. 15, no. 5, pp. 493-522, 2010

18. Professional status
vs. Expertise Related Work
Z´phyrin Soh et al.
e UML Class Diagram Comprehension (2/2)
Problem and
Motivations
Problem
Motivations
Related Work
Subjects’ categories
Expertise Studies
UML Class Diagram
Previous work used subject’s proficiency as categorisation
Comprehension
criterion:
Empirical Study
Study Design Subjects’ performance in task realization
Results
RQ1: Status
Subjects’ grade in the course they were enrolled
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
9 / 24

19. Professional status
vs. Expertise Related Work
Z´phyrin Soh et al.
e UML Class Diagram Comprehension (2/2)
Problem and
Motivations
Problem
Motivations
Related Work
Subjects’ categories
Expertise Studies
UML Class Diagram
Previous work used subject’s proficiency as categorisation
Comprehension
criterion:
Empirical Study
Study Design Subjects’ performance in task realization
Results
RQ1: Status
Subjects’ grade in the course they were enrolled
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision Motivations
Conclusion and
Future Work No previous work that uses the maintenance task on
Conclusion
Threats to Validity and
UML class diagrams and eye-tracking system to study
Future Work
separately the professional status and the expertise
Combine expertise studies and UML eye-tracking studies
9 / 24

20. Professional status
vs. Expertise Empirical Study
Z´phyrin Soh et al.
e Study Design (1/8)
Problem and
Motivations
Problem
Motivations
Related Work Research Questions
Expertise Studies
UML Class Diagram RQ1: What is the relation between a designer’s
Comprehension
Empirical Study
professional status and her class diagram
Study Design comprehension?
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
10 / 24

21. Professional status
vs. Expertise Empirical Study
Z´phyrin Soh et al.
e Study Design (1/8)
Problem and
Motivations
Problem
Motivations
Related Work Research Questions
Expertise Studies
UML Class Diagram RQ1: What is the relation between a designer’s
Comprehension
Empirical Study
professional status and her class diagram
Study Design comprehension?
Results
RQ1: Status
RQ2: What is the relation between a designer’s
RQ2: Expertise
RQ3: Status vs. Expertise
expertise and her class diagram comprehension?
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
10 / 24

22. Professional status
vs. Expertise Empirical Study
Z´phyrin Soh et al.
e Study Design (1/8)
Problem and
Motivations
Problem
Motivations
Related Work Research Questions
Expertise Studies
UML Class Diagram RQ1: What is the relation between a designer’s
Comprehension
Empirical Study
professional status and her class diagram
Study Design comprehension?
Results
RQ1: Status
RQ2: What is the relation between a designer’s
RQ2: Expertise
RQ3: Status vs. Expertise
expertise and her class diagram comprehension?
RQ4: Question Precision
RQ3: What is the most important factor between
Conclusion and
Future Work expertise and professional status?
Conclusion
Threats to Validity and
Future Work
10 / 24

23. Professional status
vs. Expertise Empirical Study
Z´phyrin Soh et al.
e Study Design (1/8)
Problem and
Motivations
Problem
Motivations
Related Work Research Questions
Expertise Studies
UML Class Diagram RQ1: What is the relation between a designer’s
Comprehension
Empirical Study
professional status and her class diagram
Study Design comprehension?
Results
RQ1: Status
RQ2: What is the relation between a designer’s
RQ2: Expertise
RQ3: Status vs. Expertise
expertise and her class diagram comprehension?
RQ4: Question Precision
RQ3: What is the most important factor between
Conclusion and
Future Work expertise and professional status?
Conclusion
Threats to Validity and
Future Work
RQ4: What is the effect of the question precision on
the comprehension of a UML class diagram?
10 / 24

24. Professional status
vs. Expertise Empirical Study
Z´phyrin Soh et al.
e Study Design (2/8)
Problem and
Motivations
Problem
Motivations
Related Work
Objects and Tasks
Expertise Studies
UML Class Diagram
Comprehension
= ArgoUML, JUnit, and QuickUML
Empirical Study
Study Design Number of Average Average
Results
classes/ number of number of
RQ1: Status Interfaces attributes per methods per
RQ2: Expertise Class/Interface Class/Interface
RQ3: Status vs. Expertise
RQ4: Question Precision
ArgoUML 10 0.4 8.6
Conclusion and JUnit 14 0.57 6.14
Future Work
Conclusion
QuickUML 16 1.75 3.87
Threats to Validity and
Future Work
= : one maintenance task per object
11 / 24

25. Professional status
vs. Expertise Empirical Study
Z´phyrin Soh et al.
e Study Design (3/8)
Problem and
Motivations
Problem
Independent variables
Motivations
Related Work = Professional status + Expertise
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
12 / 24

26. Professional status
vs. Expertise Empirical Study
Z´phyrin Soh et al.
e Study Design (3/8)
Problem and
Motivations
Problem
Independent variables
Motivations
Related Work = Professional status + Expertise
Expertise Studies
UML Class Diagram
Comprehension
Professional status
Empirical Study = practitioners (9)
Study Design = students (12)
Results
(in industry)
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
12 / 24

27. Professional status
vs. Expertise Empirical Study
Z´phyrin Soh et al.
e Study Design (3/8)
Problem and
Motivations
Problem
Independent variables
Motivations
Related Work = Professional status + Expertise
Expertise Studies
UML Class Diagram
Comprehension
Professional status
Empirical Study = practitioners (9)
Study Design = students (12)
Results
(in industry)
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
Expertise: We used the number of years of experience
RQ4: Question Precision to categorise experts and novices.
Conclusion and Pair-wise Wilcoxon comparison (+ Bonferroni
Future Work
Conclusion correction)
Threats to Validity and
Future Work Categorization with the highlest Cliff’s δ value
= experts (12): {3, 4, 5} years of experience
= novices (9): {1, 2} years of experience
12 / 24

28. Professional status
vs. Expertise Empirical Study
Z´phyrin Soh et al.
e Study Design (4/8)
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Mitigating variable
Empirical Study Question precision: The level of details in the formulation
Study Design
of the question:
Results
RQ1: Status Precise: state the kind of operation to perform
RQ2: Expertise
RQ3: Status vs. Expertise (add/remove) and the kind of target element
RQ4: Question Precision
Conclusion and
(class/method/attribute)
Future Work
Conclusion
Not precise: no operation or target element
Threats to Validity and
Future Work
13 / 24

29. Professional status
vs. Expertise Empirical Study
Z´phyrin Soh et al.
e Study Design (5/8)
Problem and
Motivations Dependent variables
Problem
Motivations Accuracy, Time spent
Related Work
Expertise Studies
Search effort = convex hull & spatial density [10]
UML Class Diagram
Comprehension Overall effort = AFD [9] and NRRF [8]
Empirical Study
Study Design Question comprehension effort = NDQA and NFQA
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision AFD: Average Fixation Duration
NRRF: Normalized Rate of Relevant Fixations
Conclusion and
Future Work NDQA: Normalized Duration in Question Area
Conclusion NFQA: Normalized Fixations in Question Area
Threats to Validity and
Future Work
[8] Jeanmart et al. (2009)
[9] Cepeda Porras and Gu´h´neuc (2010)
e e
[10] J. H. Goldberg and X. P. Kotval, Computer interface evaluation using eye
movements: methods and constructs, Journal of Industrial Ergonomics, vol.
14 / 24 24, no. 6, pp. 631-645, 1999

30. Professional status
vs. Expertise Empirical Study
Z´phyrin Soh et al.
e Study Design (6/8)
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study Convex hull area
Study Design
Smaller convex set of fixations
Results
RQ1: Status containing all subject’s fixations
RQ2: Expertise
RQ3: Status vs. Expertise Smaller convex hull ⇒ close fixations
RQ4: Question Precision
Conclusion and
⇒ less search effort
Future Work
Conclusion
Threats to Validity and
Future Work
15 / 24

31. Professional status
vs. Expertise Empirical Study
Z´phyrin Soh et al.
e Study Design (7/8)
Problem and
Motivations Spatial density
Problem
Motivations Number of visited cells / total number of cells
Related Work
Expertise Studies less visits ⇒ less search effort
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
In TAUPE [11], cell’s size = 64x64px
[11] B. D. Smet, L. Lempereur, Z. Sharafi, Y.-G. Gu´h´neuc, G. Antoniol, and
e e
N. Habra, Taupe: Visualising and analysing eye-tracking data, Science of
16 / 24 Computer Programming, 2011

32. Professional status
vs. Expertise Empirical Study
Z´phyrin Soh et al.
e Study Design (8/8)
Overall effort: Fixations’ duration and relevance
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
17 / 24

33. Professional status
vs. Expertise Empirical Study
Z´phyrin Soh et al.
e Study Design (8/8)
Question Comprehension Effort: Fixations’ count and duration
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
17 / 24

34. Professional status
vs. Expertise Results What is the relation between a
Z´phyrin Soh et al.
e RQ1: Status (1/1) designer’s professional status and her
class diagram comprehension?
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Practitioners are more accurate than students
18 / 24

35. Professional status
vs. Expertise Results What is the relation between a
Z´phyrin Soh et al.
e RQ1: Status (1/1) designer’s professional status and her
class diagram comprehension?
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Practitioners are more accurate than students
Students spent around 35% less time than
practitioners
18 / 24

36. Professional status
vs. Expertise Results What is the relation between a
Z´phyrin Soh et al.
e RQ1: Status (1/1) designer’s professional status and her
class diagram comprehension?
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Practitioners are more accurate than students
Students spent around 35% less time than
practitioners
No significant difference for other dependent variables
18 / 24

37. Professional status
vs. Expertise Results What is the relation between a
Z´phyrin Soh et al.
e RQ1: Status (1/1) designer’s professional status and her
class diagram comprehension?
Problem and
Motivations
100
100
Problem q q qq q q q q q
Motivations
80
80
Related Work
Expertise Studies
q q q q q q q q
UML Class Diagram
Accuracy (%)
Accuracy (%)
60
60
Comprehension
Empirical Study
40
40
Study Design q q
Results
20
20
RQ1: Status
RQ2: Expertise
q q
0
0
RQ3: Status vs. Expertise
RQ4: Question Precision 100 300 500 700 150 250 350
Conclusion and Time spent (s) Time spent (s)
Future Work (a) Accuracy − Time tradeoff for Practitioners (b) Accuracy − Time tradeoff for Students
Conclusion
Threats to Validity and
Future Work
Practitioners are more accurate than students
Students spent around 35% less time than
practitioners
No significant difference for other dependent variables
Students could be more accurate if spending more time
18 / 24

38. Professional status
vs. Expertise Results What is the relation between a
Z´phyrin Soh et al.
e RQ2: Expertise (1/1) designer’s expertise and her class
diagram comprehension?
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and Experts are more accurate than novices
Future Work
19 / 24

39. Professional status
vs. Expertise Results What is the relation between a
Z´phyrin Soh et al.
e RQ2: Expertise (1/1) designer’s expertise and her class
diagram comprehension?
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and Experts are more accurate than novices
Future Work
Novices spent around 33% less time than experts
19 / 24

40. Professional status
vs. Expertise Results What is the relation between a
Z´phyrin Soh et al.
e RQ2: Expertise (1/1) designer’s expertise and her class
diagram comprehension?
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and Experts are more accurate than novices
Future Work
Novices spent around 33% less time than experts
Experts have a more efficient ability to search
relevant elements than novices
19 / 24

41. Professional status
vs. Expertise Results What is the relation between a
Z´phyrin Soh et al.
e RQ2: Expertise (1/1) designer’s expertise and her class
diagram comprehension?
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and Experts are more accurate than novices
Future Work
Novices spent around 33% less time than experts
Experts have a more efficient ability to search
relevant elements than novices
No significant difference for other dependent variables
19 / 24

42. Professional status
vs. Expertise Results What is the relation between a
Z´phyrin Soh et al.
e RQ2: Expertise (1/1) designer’s expertise and her class
diagram comprehension?
Problem and
Motivations
100
100
Problem qq qq qq
q q
Motivations
80
80
Related Work
Expertise Studies
q q q q q q q q
UML Class Diagram
Accuracy (%)
Accuracy (%)
60
60
Comprehension
Empirical Study
40
40
Study Design q q
Results
20
20
RQ1: Status
RQ2: Expertise
q q
0
0
RQ3: Status vs. Expertise
RQ4: Question Precision 100 300 500 700 150 250 350
Conclusion and Time spent (s) Time spent (s)
Future Work (a) Accuracy − Time tradeoff for Experts (a) Accuracy − Time tradeoff for Novices
Conclusion
Threats to Validity and Experts are more accurate than novices
Future Work
Novices spent around 33% less time than experts
Experts have a more efficient ability to search
relevant elements than novices
No significant difference for other dependent variables
19 / 24 Novices could be more accurate if spending more time

43. Professional status
vs. Expertise Results What is the most important factor
Z´phyrin Soh et al.
e RQ3: Status vs. Expertise (1/1) between expertise and professional
status?
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results Experts are more accurate than practitioners
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
20 / 24

44. Professional status
vs. Expertise Results What is the most important factor
Z´phyrin Soh et al.
e RQ3: Status vs. Expertise (1/1) between expertise and professional
status?
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results Experts are more accurate than practitioners
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
Experts spent around 7% less time than practitioners
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
20 / 24

45. Professional status
vs. Expertise Results What is the most important factor
Z´phyrin Soh et al.
e RQ3: Status vs. Expertise (1/1) between expertise and professional
status?
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results Experts are more accurate than practitioners
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
Experts spent around 7% less time than practitioners
RQ4: Question Precision
When considering expert subjects
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
20 / 24

46. Professional status
vs. Expertise Results What is the most important factor
Z´phyrin Soh et al.
e RQ3: Status vs. Expertise (1/1) between expertise and professional
status?
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
Experts are more accurate than practitioners
RQ3: Status vs. Expertise
RQ4: Question Precision Experts spent around 7% less time than practitioners
Conclusion and When considering expert subjects
Future Work
Conclusion Experienced students are more accurate than
Threats to Validity and
Future Work experienced practitioners
20 / 24

47. Professional status
vs. Expertise Results What is the most important factor
Z´phyrin Soh et al.
e RQ3: Status vs. Expertise (1/1) between expertise and professional
status?
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
Experts are more accurate than practitioners
RQ3: Status vs. Expertise
RQ4: Question Precision Experts spent around 7% less time than practitioners
Conclusion and
Future Work
When considering expert subjects
Conclusion Experienced students are more accurate than
Threats to Validity and
Future Work experienced practitioners
Experienced students spent around 37% less time than
experienced practitioners
20 / 24

48. Professional status
vs. Expertise Results What is the most important factor
Z´phyrin Soh et al.
e RQ3: Status vs. Expertise (1/1) between expertise and professional
status?
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
Experts are more accurate than practitioners
RQ3: Status vs. Expertise
RQ4: Question Precision Experts spent around 7% less time than practitioners
Conclusion and
Future Work
When considering expert subjects
Conclusion Experienced students are more accurate than
Threats to Validity and
Future Work experienced practitioners
Experienced students spent around 37% less time than
experienced practitioners
The effects of expertise on accuracy and time depend
on the status
20 / 24

49. Professional status
vs. Expertise Results
Z´phyrin Soh et al.
e RQ4: Question Precision (1/1)
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Question Precision
Empirical Study What is the effect of the question precision on the
Study Design
comprehension of a UML class diagram?
Results
RQ1: Status The accuracy of students benefits from precise question
RQ2: Expertise
RQ3: Status vs. Expertise description
RQ4: Question Precision
Conclusion and The accuracy of novices benefits from precise question
Future Work
Conclusion
description
Threats to Validity and
Future Work
21 / 24

50. Professional status
vs. Expertise Conclusion and Future Work
Z´phyrin Soh et al.
e Conclusion (1/1)
Problem and
Motivations
Problem
Status
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
22 / 24

51. Professional status
vs. Expertise Conclusion and Future Work
Z´phyrin Soh et al.
e Conclusion (1/1)
Problem and
Motivations
Problem
Status Expertise
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
22 / 24

52. Professional status
vs. Expertise Conclusion and Future Work
Z´phyrin Soh et al.
e Conclusion (1/1)
Problem and
Motivations
Problem
Status Expertise
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Experts vs. Practitioners
22 / 24

53. Professional status
vs. Expertise Conclusion and Future Work
Z´phyrin Soh et al.
e Threats to Validity and Future Work (1/1)
Problem and
Motivations
Problem
Threats to Validity and Future Work
Motivations
Construct validity: We did not use all combination of
Related Work
Expertise Studies treatments for each system
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
23 / 24

54. Professional status
vs. Expertise Conclusion and Future Work
Z´phyrin Soh et al.
e Threats to Validity and Future Work (1/1)
Problem and
Motivations
Problem
Threats to Validity and Future Work
Motivations
Construct validity: We did not use all combination of
Related Work
Expertise Studies treatments for each system
UML Class Diagram
Comprehension
Conclusion validity: Practitioners from the same
Empirical Study
Study Design
company + difficulty to find inexperienced practitioners
Results (only one)
RQ1: Status
RQ2: Expertise
⇒ Practitioners from other company
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
23 / 24

55. Professional status
vs. Expertise Conclusion and Future Work
Z´phyrin Soh et al.
e Threats to Validity and Future Work (1/1)
Problem and
Motivations
Problem
Threats to Validity and Future Work
Motivations
Construct validity: We did not use all combination of
Related Work
Expertise Studies treatments for each system
UML Class Diagram
Comprehension
Conclusion validity: Practitioners from the same
Empirical Study
Study Design
company + difficulty to find inexperienced practitioners
Results (only one)
RQ1: Status
RQ2: Expertise
⇒ Practitioners from other company
RQ3: Status vs. Expertise
RQ4: Question Precision
Internal validity: We did not limit the time (fatigue
Conclusion and biais)
Future Work
Conclusion
⇒ Limit the experiment time to investigate how much
Threats to Validity and
Future Work time affect the subject’s accuracy
23 / 24

56. Professional status
vs. Expertise Conclusion and Future Work
Z´phyrin Soh et al.
e Threats to Validity and Future Work (1/1)
Problem and
Motivations
Problem
Threats to Validity and Future Work
Motivations
Construct validity: We did not use all combination of
Related Work
Expertise Studies treatments for each system
UML Class Diagram
Comprehension
Conclusion validity: Practitioners from the same
Empirical Study
Study Design
company + difficulty to find inexperienced practitioners
Results (only one)
RQ1: Status
RQ2: Expertise
⇒ Practitioners from other company
RQ3: Status vs. Expertise
RQ4: Question Precision
Internal validity: We did not limit the time (fatigue
Conclusion and biais)
Future Work
Conclusion
⇒ Limit the experiment time to investigate how much
Threats to Validity and
Future Work time affect the subject’s accuracy
External validity: Only three systems and small range
of years of experience
⇒ Use other systems
23 / 24

57. Professional status
vs. Expertise
Z´phyrin Soh et al.
e Thanks for your attention!
Problem and
Motivations
Problem
Status Expertise
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
The accuracy of students and novices
RQ3: Status vs. Expertise
RQ4: Question Precision
benefits from precise question descriptions
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Experts vs. Practitioners
24 / 24