Code reviews have been conducted since decades in software projects, with the aim of improving code quality from many different points of view. During code reviews, developers are supported by checklists, coding standards and, possibly, by various kinds of static analysis tools. This paper investigates whether warnings highlighted by static analysis tools are taken care of during code reviews and, whether there are kinds of warnings that tend to be removed more than others. Results of a study conducted by mining the Gerrit repository of six Java open source projects indicate that the density of warnings only slightly vary after each review. The overall percentage of warnings removed during reviews is slightly higher than what previous studies found for the overall project evolution history. However, when looking (quantitatively and qualitatively) at specific categories of warnings, we found that during code reviews developers focus on certain kinds of problems. For such categories of warnings the removal percentage tend to be very high—often above 50% and sometimes 100%. Examples of those are warnings in the imports, regular expression, and type resolution categories. In conclusion, while a broad warning detection might produce way too many false positives, enforcing the removal of certain warnings prior to the patch submission could reduce the amount of effort provided during the code review process.

1. Would Static Analysis Tools Help
Developers with Code Reviews?
Sebastiano Panichella Venera Arnaoudova Massimiliano Di Penta Giuliano Antoniol

2. OUTLINE
Context:
Code Reviews.
Case Study:
Code Reviews of
6 Open Source Projects.
Results:
Warnings Resolved by
Developers During Reviews.

3. CODE REVIEWS
Why, What, How?

4. CODE REVIEWS
Why?

5. CODE REVIEWS
Why: concrete benefits…
Improved
Code
Quality
Fewer
defects
in
Code
Improved
Knowledge
Transfer
Education
of
Junior
Programmers
Benefits
“Expectations, Outcomes, and Challenges of Modern Code Review”
Alberto Bacchelli and Christian Bird - ICSE 2013
“Common Outcomes of Code Review”

6. CODE REVIEWS
What: types of peer code reviews?
Formal
Inspection
Process
Over
The
Shoulder
Reviews
Email
Pass
Around
Interviews
Tool
assisted
reviews
Pair
Programming

7. CODE REVIEWS
What: types of peer code reviews?
Over
The
Shoulder
Reviews
Email
Pass
Around
Interviews
Tool
assisted
reviews
Pair
Programming
“Modern code review is a
form of code inspection
which has the qualities of
being informal, tool-based
and frequent.”
“Expectations, Outcomes, and Challenges of Modern Code Review”
Alberto Bacchelli and Christian Bird - ICSE 2013
Formal
Inspection
Process

8. MODERN CODE REVIEWS
“Modern code review is a
form of code inspection
which has the qualities of
being informal, tool-based
and frequent.”
“Expectations, Outcomes, and Challenges of Modern Code Review”
Alberto Bacchelli and Christian Bird - ICSE 2013

9. MODERN CODE REVIEWS: TOOLS
(I)
Code Reviews
Management

10. GERRIT: a Tool to Conduct and
Manage Code Reviews

11. GERRIT: a Tool to Conduct and
Manage Code Reviews

12. GERRIT: a Tool to Conduct and
Manage Code Reviews

13. GERRIT: a Tool to Conduct and
Manage Code Reviews

14. GERRIT: a Tool to Conduct and
Manage Code Reviews

15. MODERN CODE REVIEWS
(I)
Code Reviews
Management

16. MODERN CODE REVIEWS
(I)
Code Reviews
Management
(II)
Bugs/Issues
Detection

17. MODERN CODE REVIEWS
(I)
Code Reviews
Management
(II)
Bugs/Issues
Detection
LIMITATION:
provide a too extensive list
of recommendations

18. Past Work
Kim et al. - FSE 2007
Only10%, of suggested
warnings are removed
by bug fix changes

19. To What Extend Static Analysis Tools Help
Developers During Code Reviews?

20. To What Extent Static Analysis Tools Help
Developers During Code Reviews?
Project History

21. To What Extent Static Analysis Tools Help
Developers During Code Reviews?
Project History
During Code Reviews
We argue that the Use of
Static AnalysisTools
Would be Highly Beneficial
During Code Reviews…

22. CASE STUDY
Code Reviews of 6 Open Source Projects.

23. Goal: understanding how static analysis tools could
have helped in dealing with warnings developers solved
during code reviews.
Quality focus: reducing developers’ effort during the
code review task.
Perspective: develop tool to support the
configuration of static analysis tools towards warnings
that are considered relevant by developers.
CASE STUDY

24. RESEARCH QUESTIONS
RQ1: To what extent warnings detected by static
analysis tools are removed during code reviews?
RQ2: What kinds of warnings detected by static
analysis tool are mainly considered during code
reviews?

25. Projects Observe Period KLOC # of Reviews
Analysed
Uses
Checkstylee
Uses
PDM
Eclipse CDT
2013-11-29 -
2014-09-22
1,500–
1,550
309
Eclipse Platform
UI
2013-06-24 -
2014-09-09
2,092–
2,305
16
Eclipse JDT
Core
2013-05-23 -
2014-09-24
2,736–
2,554
113
OpenDaylight
Controller
2013-01-01 -
2014-09-24
149–171 161
Motech
2013-07-24 -
2014-09-24
586–1,909 209
Vaadin 2013-06-01 -
2014-09-24
6,174–
6,114
180
CONTEXT
Object:
Tools Experimented:

26. STUDY PROCEDURE

27. PATCH SETS COMPARISON…
Given a
Code Review

28. PATCH SETS COMPARISON…
Given a
Code Review
We use…

29. PATCH SETS COMPARISON…
Given a
Code Review
We use... to compare
warnings density
variation between…
Firstpatch
Lastpatch

31. RQ1
To what extent warnings detected by
static analysis tools are removed
during code reviews?

32. Projects
Density of Warnings
[P-value]
# of Warning
[P-value]
Density of Warnings
[P-value]
# of Warning
[P-value]
Eclipse CDT 0.074 0.025 0.028 <001
Eclipse JDT Core 0.450 0.919 0.351 0.624
Eclipse Platform
UI
0.132 0.857 0.011 0.2
OpenDaylight
Controller
0.080 <0.01 0.614 <0.01
Motech >0.01 <0.01 0.205 <0.01
Vaadin NA NA 0.148 0.209
Changes of Warnings Density (and Absolute
Number) During Code Reviews.

33. Projects
Density of Wornings
[P-value]
# of Warning
[P-value]
Density of Wornings
[P-value]
# of Warning
[P-value]
Eclipse CDT 0.074 0.025 0.028 <001
Eclipse JDT Core 0.450 0.919 0.351 0.624
Eclipse Platform
UI
0.132 0.857 0.011 0.2
OpenDaylight
Controller
0.080 <0.01 0.614 <0.01
Motech >0.01 <0.01 0.205 <0.01
Vaadin NA NA 0.148 0.209
Changes of Warnings Density (and Absolute
Number) During Code Reviews.

34. Projects
Density of Wornings
[P-value]
# of Warning
[P-value]
Density of Wornings
[P-value]
# of Warning
[P-value]
Eclipse CDT 0.074 0.025 0.028 <001
Eclipse JDT Core 0.450 0.919 0.351 0.624
Eclipse Platform
UI
0.132 0.857 0.011 0.2
OpenDaylight
Controller
0.080 <0.01 0.614 <0.01
Motech >0.01 <0.01 0.205 <0.01
Vaadin NA NA 0.148 0.209
Changes of Warnings Density (and Absolute
Number) During Code Reviews.

35. Cumulative Percentage of
Removed Warnings
Projects
Uses
Checkstyle
Uses
PDM
% of Resolved Warnings % of Resolved Warnings
Eclipse CDT 11% 11%
Eclipse
Platform UI
5% 7%
Eclipse JDT
Core
11% 9%
OpenDaylight
Controller
15% 15%
Motech 23% 13%
Vaadin - 13%

36. Cumulative Percentage of
Removed Warnings
Projects
Uses
Checkstyle
Uses
PDM
% of Resolved Warnings % of Resolved Warnings
Eclipse CDT 11% 11%
Eclipse
Platform UI
5% 7%
Eclipse JDT
Core
11% 9%
OpenDaylight
Controller
15% 15%
Motech 23% 13%
Vaadin - 13%

37. RQ2
What kinds of warnings detected by
static analysis tool are mainly
considered during code reviews?

38. Qualitative Analysis

39. Qualitative Analysis

40. Qualitative Analysis

41. Qualitative Analysis
“We randomly sampled 10% of code
reviews that resolved at least one warning”

42. Qualitative AnalysisQualitative Analysis
“Warning that Developers
Fix During Code Reviews:”

43. Qualitative Analysis
“Warning that Developers
Fix During Code Reviews:”
Type Resolution

44. Qualitative Analysis
“Warning that Developers
Fix During Code Reviews:”
Unused code
Type Resolution

45. Qualitative Analysis
“Warning that Developers
Fix During Code Reviews:”
Imports
Regular Expression
Type Resolution
Unused code

46. Qualitative Analysis
“Warning that Developers
Fix During Code Reviews:”
Imports
Regular Expression
Type Resolution
Unused code

47. Eclipse CDT:
Percentage of PDM’ Resolved Warnings
Warning Types % Resolved Warnings
Type Resolution 100%
Import 100%
Basic 75%
Sunsecure 67%
Codesize 59%
Unusedcode 58%
Logging-java 51%
j2ee 47%
Design 42%
junit 38%
Empty 33%
Javabeans 26%
Naming 14%
Braces 14%
…. …..

48. Eclipse CDT:
Percentage of PDM’ Resolved Warnings
Warning Types % Resolved Warnings
Type Resolution 100%
Import 100%
Basic 75%
Sunsecure 67%
Codesize 59%
Unusedcode 58%
Logging-java 51%
j2ee 47%
Design 42%
junit 38%
Empty 33%
Javabeans 26%
Naming 14%
Braces 14%
…. …..
“Quantitative Analisys Confirms
Findings of the Qualitative analysis..”

49. OpenDaylight Controller:
Percentage of Checkstyle’ Resolved Warnings
Warning Types % Resolved Warnings
Regular Expressions 100%
Modifiers 100%
Metrics 100%
import 53%
Whitespace 48%
Class Design 47%
Annotations 40%
Naming 16%
Coding 15%
%Javadoc Comments 12%
Size Violations 11%
Javabeans 26%
Block Checks 10%
Miscellaneous 8%
…. …..
“Similar Results for Checkstyle Warnings..”

50. OpenDaylight Controller:
Percentage of Checkstyle’ Resolved Warnings
Warning Types % Resolved Warnings
Regular Expressions 100%
Modifiers 100%
Metrics 100%
import 53%
Whitespace 48%
Class Design 47%
Annotations 40%
Naming 16%
Coding 15%
%
Javadoc Comments 12%
Size Violations 11%
Javabeans 26%
Block Checks 10%
Miscellaneous 8%
…. …..
Developers Fix also
Warnings related to:
1) naming convention
2) code formatting
3) code comments

51. By implication…
“Enforcing the removal of certain warnings
before submitting a patch..”