1.
Mining Co-Change Information
to Understand when Build
Changes are Necessary
@shane_mcintosh
Shane
McIntosh
Bram
Adams
Meiyappan
Nagappan
Ahmed E.
Hassan

2.
What is a build system?
Source
code
2

3.
What is a build system?
Source
code
Deliverable
2

4.
Build systems describe how sources are!
.c
.cc
.tex
.o
.o
.dvi
.a
.exe
.pdf
.deb
translated into deliverables
3

5.
Continuous Integration:!
Enabled by the
build system
4
.c .mk

6.
Continuous Integration:!
Enabled by the
build system
Commit
4
Commit
9719cf0
.c .mk

7.
Continuous Integration:!
Enabled by the
build system
Commit
4
Build
Commit
9719cf0
.c .mk

8.
Continuous Integration:!
Enabled by the
build system
Commit
4
Build
Test
Commit
9719cf0
.c .mk

9.
Commit
9719cf0
Continuous Integration:!
Enabled by the
build system
Commit
4
Build
Test
Report
Commit
9719cf0 was
successfully
integrated
.c .mk

10.
Commit
9719cf0
Continuous Integration:!
Enabled by the
build system
Commit
4
Build
Test
Report
Commit
9719cf0 was
successfully
integrated
.c .mk

11.
“...nothing can be
said to be certain,
except death and
taxes”
- Benjamin Franklin
The Build “Tax”
Up to 27% of source
changes require build
changes, too!
An Empirical Study of Build
Maintenance Effort!
S. McIntosh, B. Adams, T. H. D.
Nguyen, Y. Kamei, A. E. Hassan
[ICSE 2011]
5

12.
Neglected build maintenance!
is a frequent cause build breaks
6
.c .mk

13.
Commit
Neglected build maintenance!
is a frequent cause build breaks
6
Commit
aedd38
.c
.mk

14.
Commit
Neglected build maintenance!
is a frequent cause build breaks
6
Commit
aedd38
.c
.mk

15.
Commit
Neglected build maintenance!
is a frequent cause build breaks
6
Build
Commit
aedd38
.c
.mk

16.
Commit
Neglected build maintenance!
is a frequent cause build breaks
6
Build
Test
Commit
aedd38
.c
.mk

17.
Commit
Neglected build maintenance!
is a frequent cause build breaks
6
Build
Test
Commit
aedd38
.c
.mk

18.
Commit
Neglected build maintenance!
is a frequent cause build breaks
6
Build
Test
Report
Commit
aedd38
.c
.mk
Commit
aedd38
broke the
build!

19.
Neglected build maintenance!
can even impact end users
7

20.
Neglected build maintenance!
can even impact end users
7
Not working due
to linking of
incorrect SQLite
library version

21.
Neglected build maintenance!
can even impact end users
7
When are build!
changes necessary?
Not working due
to linking of
incorrect SQLite
library version

22.
Overview of the studied systems
8
C++ Java

23.
Overview of the studied systems
8
C++ Java

24.
Overview of the studied systems
8
C++ Java
13 years of
historical data

25.
Overview of the studied systems
8
C++ Java
13 years of
historical data
Single build system for
Firefox, Thunderbird, etc.

26.
Overview of the studied systems
8
C++ Java
13 years of
historical data
Single build system for
Firefox, Thunderbird, etc.
Total of 16 years of
historical data

27.
Overview of the studied systems
8
C++ Java
13 years of
historical data
Single build system for
Firefox, Thunderbird, etc.
Total of 16 years of
historical data
Proprietary and open
source systems

28.
Semi-automatic identification of file
types using naming conventions
.m..cck ..ad.caht
9

29.
Semi-automatic identification of file
types using naming conventions
.c .mk
.c
Source Build
Test
.h .dat .ac
Source Test Build
9

30.
Grouping related changes according
to the work items that they address
10

31.
Grouping related changes according
to the work items that they address
.c .c .c
Changes
.mk
10

32.
Grouping related changes according
to the work items that they address
Missed code!
in #2121
.c .c .c
Add feature!
#2121
Fix for!
bug #1234
Changes
Transactions
.mk
10

33.
Grouping related changes according
to the work items that they address
Missed code!
2121
in #2121
.c .c .c
Add feature!
#2121
Fix for!
bug #1234
1234
Changes
Transactions
Work items
.mk
10

34.
11

35.
!
Case study structure
11

36.
!
Case study structure
11
(RQ1)!
Co-change
frequency
.c ? .mk

37.
!
Case study structure
(RQ2)!
Co-change
prediction
accuracy
11
(RQ1)!
Co-change
frequency
.c ? .mk

38.
!
Case study structure
(RQ2)!
Co-change
prediction
accuracy
11
(RQ1)!
Co-change
frequency
.c ? .mk
(RQ3)!
Most powerful
co-change
indicators

39.
!
Case study structure
(RQ2)!
Co-change
prediction
accuracy
12
(RQ1)!
Co-change
frequency
.c ? .mk
(RQ3)!
Most powerful
co-change
indicators

40.
The proportion of build changes that are
accompanied by source code changes
13
.c ? .mk

41.
The proportion of build changes that are
accompanied by source code changes
.mk .c .c .mk .mk .c
13
.c ? .mk

42.
The proportion of build changes that are
accompanied by source code changes
13
.c ? .mk
.c .c
Work!
items
.mk
1
.mk .mk .c
2 3

43.
The proportion of build changes that are
accompanied by source code changes
13
.c ? .mk
.c .c
Work!
items
.mk
1
.mk .mk .c
2 3
2 build co-changes
3 total build changes

44.
Build changes are often accompanied
by changes to the source code
Mozilla Eclipse Lucene Jazz
Build ⇒ Source 86% 82% 44% 72%
Build ⇒ Test 29% 36% 41% 36%
Build ⇒ Src/Test 88% 82% 53% 77%
14
.c ? .mk

45.
Build changes are occasionally
accompanied by changes to test code
Mozilla Eclipse Lucene Jazz
Build ⇒ Source 86% 82% 44% 72%
Build ⇒ Test 29% 36% 41% 36%
Build ⇒ Src/Test 88% 82% 53% 77%
15
.c ? .mk

46.
53%-88% of build changes are accompanied
by changes to source or test code
Mozilla Eclipse Lucene Jazz
Build ⇒ Source 86% 82% 44% 72%
Build ⇒ Test 29% 36% 41% 36%
Build ⇒ Src/Test 88% 82% 53% 77%
16
.c ? .mk

47.
!
Case study structure
(RQ2)!
Co-change
prediction
accuracy
17
(RQ1)!
Co-change
frequency
.c ? .mk
(RQ3)!
Most powerful
co-change
indicators
Most build
changes are
accompanied
by source/test
code changes

48.
!
Case study structure
(RQ2)!
Co-change
prediction
accuracy
18
(RQ1)!
Co-change
frequency
.c ? .mk
(RQ3)!
Most powerful
co-change
indicators
Most build
changes are
accompanied
by source/test
code changes

49.
Metrics that may indicate whether a code change
will require an accompanying build co-change
19

50.
Metrics that may indicate whether a code change
will require an accompanying build co-change
19
Language-agnostic

51.
Metrics that may indicate whether a code change
will require an accompanying build co-change
19
Language-agnostic
+++ File added

52.
Metrics that may indicate whether a code change
will require an accompanying build co-change
19
Language-agnostic
+++ File added
Historical .c
.mk
tendencies

53.
Metrics that may indicate whether a code change
will require an accompanying build co-change
Language-agnostic
Language-aware
19
+++ File added
Historical .c
.mk
tendencies

54.
Metrics that may indicate whether a code change
will require an accompanying build co-change
Language-agnostic
Language-aware
19
+++ File added
Historical .c
.mk
tendencies
Changed
dependencies
import

55.
Metrics that may indicate whether a code change
will require an accompanying build co-change
Language-agnostic
Language-aware
19
+++ File added
Historical .c
.mk
tendencies
Changed
dependencies
import
Changed!
conditional
compilation
#ifdef

56.
We train classifiers to identify code
changes that require build changes
.mk
1 2
20
Work!
items
.c .c .c
Random forest
classification model
Build change
necessary
No build change
necessary

57.
We train classifiers to identify code
changes that require build changes
.mk
1 2
20
Work!
items
.c
Ran.dcom .fcorest
classification model
Build change
necessary
No build change
necessary

58.
We train classifiers to identify code
changes that require build changes
.mk
1 2
21
Work!
items
.c
Random forest
classification model
Build change!
necessary
No build change
necessary

59.
We train classifiers to identify code
changes that require build changes
.mk
1 2
22
Work!
items
.c .c .c
Random forest
classification model
Build change
necessary
No build change
necessary

60.
We train classifiers to identify code
changes that require build changes
.mk
1 2
22
Work!
items
.c
.c .c
Random forest
classification model
Build change
necessary
No build change
necessary

61.
We train classifiers to identify code
changes that require build changes
.mk
1 2
23
Work!
items
.c .c
Random forest
classification model
Build change
necessary
No build change!
necessary

62.
Language-aware metrics add
significant explanatory power
Mozilla Eclipse Lucene Jazz
24
AUC (language-agnostic
metrics)
0.84 0.66 0.71 0.57
AUC (language-agnostic
and
aware metrics)
0.88 0.69 0.78 0.60

63.
Mozilla classifiers are highly
accurate
Mozilla Eclipse Lucene Jazz
25
AUC (language-agnostic
metrics)
0.84 0.66 0.71 0.57
AUC (language-agnostic
and
aware metrics)
0.88 0.69 0.78 0.60

64.
Java classifiers are less accurate
than Mozilla one
Mozilla Eclipse Lucene Jazz
26
AUC (language-agnostic
metrics)
0.84 0.66 0.71 0.57
AUC (language-agnostic
and
aware metrics)
0.88 0.69 0.78 0.60

65.
Java build changes are rarely!
driven by code changes
Category Task Amount Correctly
System structure Refactorings 19 (25%) class8ified
General build
27
maintenance
Build tool
configuration
15 (20%) 0
Build defects 6 (8%) 0
Release
engineering
Add platform
support
12 (16%) 2
Packaging fixes 12 (16%) 3
Library
versioning
8 (11%) 0
Test maintenance Test
infrastructure
3 (4%) 0

66.
!
Case study structure
(RQ2)!
Co-change
prediction
accuracy
28
(RQ1)!
Co-change
frequency
.c ? .mk
(RQ3)!
Most powerful
co-change
indicators
Most build
changes are
accompanied
by source/test
code changes
AUCC++ 0.88,
AUCJava 0.60-0.88
Java build changes
are rarely motivated
by code changes

67.
!
Case study structure
(RQ2)!
Co-change
prediction
accuracy
29
(RQ1)!
Co-change
frequency
.c ? .mk
(RQ3)!
Most powerful
co-change
indicators
Most build
changes are
accompanied
by source/test
code changes
AUCC++ 0.88,
AUCJava 0.60-0.88
Java build changes
are rarely motivated
by code changes

68.
Measuring variable importance in
random forest classifiers
30
Var1 Var2
500
Random Forests!
L. Breiman
[Machine Learning 2001]
150
Build!
change?
No
No
Yes
Var3
2
456
777
111
65
4
13
Random forest
classification model
Classification
Work!
Item
1
2
3

69.
Measuring variable importance in
random forest classifiers
30
Var1 Var2
500
Random Forests!
L. Breiman
[Machine Learning 2001]
150
Build!
change?
No
No
Yes
Var3
2
456
777
111
65
4
13
Random forest
classification model
Classification
Work!
Item
1
2
3
No
Yes
Yes

70.
Measuring variable importance in
random forest classifiers
30
Var1 Var2
500
Random Forests!
L. Breiman
[Machine Learning 2001]
150
Build!
change?
No
No
Yes
Var3
2
456
777
111
65
4
13
Random forest
classification model
Classification
Work!
Item
1
2
3
No
Yes
Yes
Misclassification rate of
1 ÷ 3 = 0.33

71.
Measuring variable importance in
random forest classifiers
30
Var1 Var2
500
Random Forests!
L. Breiman
[Machine Learning 2001]
150
Build!
change?
No
No
Yes
Var3
2
456
777
111
65
4
13
Random forest
classification model
Classification
Work!
Item
1
2
3
No
Yes
Yes
Randomly
permute values

72.
Measuring variable importance in
random forest classifiers
31
Var1 Var2
777
500
Random Forests!
L. Breiman
[Machine Learning 2001]
150
Build!
change?
No
No
Yes
Var3
2
456
111
65
4
13
Random forest
classification model
Classification
Work!
Item
1
2
3

73.
Measuring variable importance in
random forest classifiers
31
Var1 Var2
777
500
Random Forests!
L. Breiman
[Machine Learning 2001]
150
Build!
change?
No
No
Yes
Var3
2
456
111
65
4
13
Random forest
classification model
Classification
Work!
Item
1
2
3
Yes
Yes
Yes

74.
Measuring variable importance in
random forest classifiers
31
Var1 Var2
777
500
Random Forests!
L. Breiman
[Machine Learning 2001]
150
Build!
change?
No
No
Yes
Var3
2
456
111
65
4
13
Random forest
classification model
Classification
Work!
Item
1
2
3
Yes
Yes
Yes
Misclassification rate of
2 ÷ 3 = 0.67

75.
Measuring variable importance in
random forest classifiers
Var1 randomness increases
misclassification rate by 0.33
31
Var1 Var2
777
500
Random Forests!
L. Breiman
[Machine Learning 2001]
150
Build!
change?
No
No
Yes
Var3
2
456
111
65
4
13
Random forest
classification model
Classification
Work!
Item
1
2
3
Yes
Yes
Yes
Misclassification rate of
2 ÷ 3 = 0.67

76.
Variable importance in the!
Mozilla classifier
Mozilla Eclipse−core ●
●
●
●
● ● ●●
●●
0.10
0.05
0.00
Non−core dependencies
Conditional compilation
Test deleted
Source modified
Test added
Source deleted
Source added
Prior build co−changes
Test modified
Source renamed
Test renamed
Test added
Source deleted
Source added
Number of files
Non−core dependencies
Prior build co−changes
Test deleted
Source modified
Test modified
Source renamed
Number of files
Test renamed
Source added
Test Source Variable Importance Score
32

77.
Variable importance in the!
Mozilla classifier
Mozilla Eclipse−core ●
●
●
●
● ● ●●
●●
0.10
0.05
0.00
Non−core dependencies
Conditional compilation
Test deleted
Source modified
Test added
Source deleted
Source added
Prior build co−changes
Test modified
Source renamed
Test renamed
Test added
Source deleted
Source added
Number of files
Non−core dependencies
Prior build co−changes
Test deleted
Source modified
Test modified
Source renamed
Number of files
Test renamed
Source added
Test Source Variable Importance Score
32
Structure-altering
changes are
most important in
C++ classifiers

78.
Variable importance in the!
Mozilla classifier
Mozilla Eclipse−core ●
●
●
●
● ● ●●
●●
0.10
0.05
0.00
Non−core dependencies
Conditional compilation
Test deleted
Source modified
Test added
Source deleted
Source added
Prior build co−changes
Test modified
Source renamed
Test renamed
Test added
Source deleted
Source added
Number of files
Non−core dependencies
Prior build co−changes
Test deleted
Source modified
Test modified
Source renamed
Number of files
Test renamed
Source added
Test Source Variable Importance Score
32
Structure-altering
changes are
most important in
C++ classifiers
Historical co-change
tendencies
are also important

79.
Variable importance in Java
classifiers
Eclipse−core Lucene Jazz
Non−core dependencies
Prior build co−changes
Test deleted
Source modified
33
● ●●
●●
●
●
●
Non−core dependencies
Prior build co−changes
Test deleted
Source modified
Test modified
Source renamed
added
Test added
Source deleted
Test added
Source deleted
Number of files
Test renamed
Source added
Test modified
Source renamed
Non−core dependencies
Test deleted
Source modified
Test added
Source deleted
Number of files
Test renamed
Source added
Test modified
Source renamed
Test renamed

80.
Variable importance in Java
classifiers
Eclipse−core Lucene Jazz
Non−core dependencies
Prior build co−changes
Test deleted
Source modified
33
● ●●
●●
●
●
●
Non−core dependencies
Prior build co−changes
Test deleted
Source modified
Test modified
Source renamed
added
Test added
Source deleted
Test added
Source deleted
Number of files
Test renamed
Source added
Test modified
Source renamed
Non−core dependencies
Test deleted
Source modified
Test added
Source deleted
Number of files
Test renamed
Source added
Test modified
Source renamed
Test renamed
Structure-altering is less
important than in Mozilla

81.
Variable importance in Java
classifiers
Eclipse−core Lucene Jazz
Non−core dependencies
Prior build co−changes
Test deleted
Source modified
33
● ●●
●●
●
●
●
Non−core dependencies
Prior build co−changes
Test deleted
Source modified
Test modified
Source renamed
added
Test added
Source deleted
Test added
Source deleted
Number of files
Test renamed
Source added
Test modified
Source renamed
Non−core dependencies
Test deleted
Source modified
Test added
Source deleted
Number of files
Test renamed
Source added
Test modified
Source renamed
Test renamed
Structure-altering is less
important than in Mozilla
Java classifiers rely more
on historical co-change
tendencies

82.
!
Case study structure
(RQ2)!
Co-change
prediction
accuracy
34
(RQ1)!
Co-change
frequency
.c ? .mk
(RQ3)!
Most powerful
co-change
indicators
Most build
changes are
accompanied
by source/test
code changes
AUCC++ 0.88,
AUCJava 0.60-0.88
Java build changes
are rarely motivated
by code changes
C++: Structure-altering
changes
Java: Historical
co-change
tendencies