During the past ten years, static analysis tools have become a vital part of software development for many organizations. However, the question arises, “Can we quantify the benefits of static analysis?” William Oliver presents the results of a Lawrence Livermore National Laboratory study that first measured the cost of finding software defects using formal testing on a system without static analysis; then, they integrated a static analysis tool into the process and, over a period of time, recalculated the cost of finding software defects. Join William as he shares the results of their study and discusses the value and benefits of static testing. Learn how commercial and open source analysis tools can perform sophisticated source code analysis over large code bases. Take back proof that employing static analysis can not only reduce the time and cost of finding defects and their subsequent debugging but ultimately can reduce the number of defects making their way into your releases.

1. Lawrence Livermore National Laboratory
Quantifying the Value of Static
Analysis
Date 5/19/2011
William B. Oliver
Lawrence Livermore National Laboratory, P. O. Box 808, Livermore, CA 94551
This work performed under the auspices of the U.S. Department of Energy by
Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344
LLNL-PRES-490136

2. What is Static Analysis
 The use of tools during pre-testing to remove structural
defects
Software Developer
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
2

3. What is Static Analysis
 Static analysis tools provide an in-depth analysis of
source code to find defects.
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
3

4. What is Static Analysis
 Finds Defects that compilers and traditional testing miss
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
4

5. What is Static Analysis
 Defect types include but not limited to
• Use of uninitialized variables
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
5

6. What is Static Analysis
 Defect types include but not limited to
• Use of uninitialized variables
• Memory leaks
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
6

7. What is Static Analysis
 Defect types include but not limited to
• Use of uninitialized variables
• Memory leaks
• Null Pointer dereferences
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
7

8. What is Static Analysis
 Defect types include but not limited to
• Use of uninitialized variables
• Memory leaks
• Null Pointer dereferences
• Array Bounds Overflows (and many others)
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
8

9. Why Incorporate Static Analysis
 Static Analysis finds additional defects
 Better Code Coverage
 Reduced Developer Debug Time
 Advanced Tools find Defects Inter-Procedurally
 Uncovers structural defects that can cause Functional
Defects
 Finds defects missed during code reviews/walk thrus
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
9

10. Inter-Procedural Analysis
File A
foo(x)
File B
bar(y)
File C
foobar(z)
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
10

11. Why Incorporate Static Analysis
Enhances Dynamic Testing
• Dynamic testing does not
generally uncover memory
leaks and other structural
defects
• Static Analysis provides 100 %
code coverage
• Structural defects such as
Array Bounds Overflows can
cause Functional Defects
“They are best used in combination
with traditional dynamic testing
techniques, and can even reduce the
cost to create and manage test cases
for stringent run-time coverage..”
Dr. Paul Anderson PhD
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
11

12. Structural Defects vs Functional Defects
Relate to conformance to the
Programming Language rules and
syntax
SD
FD
Uninitialized Data
Memory/Resource Leaks
Array Bounds Overflows
Null Pointer Dereferences
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
12

13. Structural Defects vs Functional Defects
Associated with Features,
Performance, Availability
etc.
SD
FD
Found During Dynamic Testing
Some Causes Include:
Solving the wrong problem
Code Logic Errors
System Integration Issues
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
13

14. Uninitialized Data
Correct Set of Values
Set of Right Answers
y
z
x
Random Set of Values
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
14

15. Memory/Resource Leaks
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
15

16. Array Bounds Overflow
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
16

17. NULL Pointer Dereference
This memory location contains the address of this memory location
Pointer
Value of the contents of address pointed to by the pointer
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
17

18. NULL Pointer Dereference
Pointer = NULL
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
18

19. Typical Static Analysis Work Flow
Perform
Static
Analysis
Add
Features
Test Fixes
Analyze
Defects
Fix Defects
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
19

20. Advantages for Testers
o Less Wasted Time
o Allows more time for test case development
o Better Test Cases
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
20

21. Assumptions About Time to Find Defects
One Million Lines of Code
Static Analysis
1000 Defects
20 Percent False Positives
800 Valid Defects
Time to Run Code Thru Tool Negligable
Ten Minutes Per Defect to Triage
Dynamic Testing
Automated Testing: 1 hour per defect
Includes Test Case Development
Test Evaluation
Test Report Generation
Manual Testing: 2 hours per Defect
1000 Defects
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
21

22. Automated Testing
TD = Total Defects = SD + FD
Time = Time to Find SD + Time to find FD
TD = 800 + 1000 = 1800
SD Time = (1000 Defects * 10 min/defect)/60min/hour = 166.67 hours
Time = 166.67 + 1000 = 1166.67 hours
Time/TD = 1166.67/1800 = .65 hours/defect = 39 minutes per defect
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
22

23. Test Case: Automated Testing
Code Type:
Programming Language:
Number of Developers:
Source Lines of Code Analyzed:
Scientific Simulation
C++
4
161,880
Total Number SD found:
528
Total Number SD Analyzed:
190
Number of False Positives:
55
Average Analysis Time/Defect:
8.9 minutes
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
23

24. Test Case: Automated Testing
TD = Total Defects = SD + FD
FD = 297 for dynamic testing and 1 hour per defect
TD = 135 + 297 = 432
SD Time = (190 Defects * 8.9 min/defect)/60min/hour = 28 hours
Time = 28 + 297 = 325 hours
Time/TD = 325/432 = .75 hours/defect = 45 minutes per defect
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
24

25. Just For Fun What If All 528 defects were triaged
Assuming 28 % False Positive Rate
Total Number SD found:
528
Total Number SD Analyzed:
528
Number of False Positives:
148
Average Analysis Time/Defect:
8.9 minutes
Estimated number of real defects = 380
Estimated Time = (528 * 8.9) / 60 = 78 hours
TD = 380 + 297 = 677
Time = 78 + 297 = 375 hours
Time/TD = 375/677 = .55 hours/defect = 33 minutes per defect
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
25

26. Manual Testing
Code Type:
Programming Language:
Security Access
C#
Number of Developers:
Total Number SD found:
76
Total Number SD Analyzed:
35
Number of False Positives:
0
Average Analysis Time/Defect:
3.4 minutes
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
26

27. Manual Testing
FD = 339 for dynamic testing and 5 hours per defect
TD = 35 + 339 = 374
SD Time = (35 Defects * 3.4 min/defect)/60min/hour = 2 hours
Time = 2 + 1695 = 1697 hours
Time/TD = 1697/374 = 4.5 hours/defect
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
27

28. Bottom Line
Combined with dynamic testing
Static Analysis results in finding
more Defects
And the organization spends less
time per defect in the process
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
28

29. Summary
For Static Analysis the time to Find
a defect is less than or equal to 10
minutes
Dynamic Testing:
Automated: 1 hour per Defect
Manual: 4 – 5 hours per Defect
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
29

30. New Breed of Tester
Perform
Static
Analysis
Analyze
Defects
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
30

31. Questions???
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
31