Student Research Competition presentation at FSE 2015.

1. Christopher Theisen
Automated Attack Surface
Approximation

2. 1/11

3. Background
Attack Surface?
Ex. early approximation of attack surface – Manadhata [1]:
Only covers API entry points
…easy to say, hard to define (practically).
OWASP defines Attack Surface as the paths in and
out of a system, the data that travels those paths,
and the code that protects both
2/11
[1] Manadhata, P., Wing, J., Flynn, M., & McQueen, M. (2006, October). Measuring the attack surfaces of two FTP daemons. In Proceedings of the 2nd
ACM workshop on Quality of protection (pp. 3-10). ACM

4. The goal of this research is to aid
software engineers in prioritizing
security efforts by approximating the
attack surface of a system via crash
dump stack trace analysis.
3/11

5. Proposed Solution
Crashes represent user activity that
puts the system under stress
We *know* external input touched the
entities on the stack trace
Are there security implications?
H1: Crash dumps localize vulnerabilities
4/11
foo!foobarDeviceQueueRequest+0x68
foo!fooDeviceSetup+0x72
foo!fooAllDone+0xA8
bar!barDeviceQueueRequest+0xB6
bar!barDeviceSetup+0x08
bar!barAllDone+0xFF
center!processAction+0x1034
center!dontDoAnything+0x1030

6. Overview
Catalog all code that appears on stack traces
5/11

7. Overview
Catalog all code that appears on stack traces
5/11

8. Overview
Catalog all code that appears on stack traces
5/11

9. Attack Surface Analysis
Windows 8 [2] Fuzzing User Crashes*
%binaries 0.9% 48.4%
%vulnerabilities 14.9% 94.6%
*Stack traces from dogfood testing crashes and field crashes
6/11
[2] C. Theisen, K. Herzig, P. Morrison, B. Murphy, and L. Williams, “Approximating Attack Surfaces with Stack Traces,” in Companion
Proceedings of the 37th International Conference on Software Engineering, 2015
Mozilla Firefox User Crashes
%files 8.4%
%vulnerabilities 72.1%
Stack traces highlighted where
security vulnerabilities were.

10. Vulnerability Prediction Models
Generate VPM based on 29 metrics (Churn, LoC, etc.) [3]
Run the VPM with all files considered as possibly vulnerable
Repeat, but remove code not found on stack traces
Vulnerability Prediction Model (VPM)
Precision improved from 0.5 to 0.69
Recall improved from 0.02 to 0.05
Statistical improvement? Yes. Practical? No.
Results [2]
[3] T. Zimmermann, N. Nagappan and L. Williams, "Searching for a Needle in a Haystack: Predicting Security Vulnerabilities for Windows Vista," in
Software Testing, Verification and Validation (ICST), 2010 Third International Conference on, 2010
7/11
[2] C. Theisen, K. Herzig, P. Morrison, B. Murphy, and L. Williams, “Approximating Attack Surfaces with Stack Traces,” in Companion
Proceedings of the 37th International Conference on Software Engineering, 2015

11. Firefox Analysis
More crashes = more vulnerabilities?
More stack traces, less files, higher flaw density!
Lose coverage as you increase stack trace cutoff
Priority: Bottom up
Introduction | Methodology | Results and Discussion | Future Work | Conclusion
Files Flaws %Files %Vuln Precision Recall
>= 1 4998 282 8.4% 72.1% 0.056 0.721
>= 30 1853 210 3.1% 53.7% 0.113 0.537
>= 140 969 162 1.6% 41.4% 0.167 0.414
All 59437 391 - - - -
8/11

12. Future Work
Introduction | Methodology | Results and Discussion | Future Work | Conclusion 9/11
Temporal Analysis
Initial attack surface approximation ...old nodes removed, new nodes added
Are new files now on the attack surface?
Are legacy files files now on the attack surface?
Preliminary: Win 10 files dropped over time, but
(old) items added back!

13. Future Work
Introduction | Methodology | Results and Discussion | Future Work | Conclusion 10/11
Few to Many Many to Many Many to Few
What are the security impact of these shapes?
Preliminary: 65% of entities have less than 5 links
Shape Analysis
A A
A

14. Introduction | Methodology | Results and Discussion | Future Work | Conclusion
foo!foobarDeviceQueueRequest+0x68
foo!fooDeviceSetup+0x72
foo!fooAllDone+0xA8
bar!barDeviceQueueRequest+0xB6
bar!barDeviceSetup+0x08
bar!barAllDone+0xFF
center!processAction+0x1034
center!dontDoAnything+0x1030
Thanks to…
11/11
Laurie Williams
Brendan Murphy
Kim Herzig
Windows Product Teams
…and many more