Slides for my ISSA Ottawa Chapter talk. The following is the link to the talk: https://issa-ottawa.ca/meetinginfo.php?id=62&ts=1542933178

1. Why Johnny Still Can’t
Pentest:
A Comparative Analysis of Open-source Black-box
Web Application Vulnerability Scanners
@rana__khalil
Rana Khalil, University of Ottawa

2. Who am I?
• Student at the University of Ottawa
• B.S. in Mathematics and Computer
Science (2016)
• M.S. in Computer Science (2018)
• Supervisor: Dr. Carlisle Adams
• OSCP Certification (current)
• Previous work experience include:
software development, testing,
ransomware research, teaching and
penetration testing
2

3. Agenda
1. Introduction
2. Methodology
3. Results
4. Conclusion
3

4. Introduction

5. Web Applications
• We use web applications for
everything:
• Over 3.9 billion users world wide
• Over 1.8 billion websites online
5
Banking Education
Shopping Communication
• How much personal data do you have
online?
• Name, SIN, addresses, phone numbers,
emails
• Financial information
• Heath information

6. Web Security
• State of web security today
• Trustwave’s 2018 Global Security
Report:
• 100% of web applications displayed
at least one vulnerability
• Median number of 11 vulnerabilities
per application
6

7. Data Breaches
7

8. How to Secure a Web Application?
• A combination of techniques are
used to secure web applications:
8
• Static code analysis
• Web application firewalls
• Secure coding practices
• Web application vulnerability scanners

9. How to Secure a Web Application?
• A combination of techniques are
used to secure web applications.
9
• Static code analysis
• Web application firewalls
• Secure coding practices
• Web application vulnerability scanners

10. WAVS
Web Application Vulnerability Scanners have three modules:
10
Crawler Attacker Analysis
*XSS found*
*SQLi found*
*LFI found*
*RFI found*

11. WAVS
Web application vulnerability scanners are largely used in two ways:
1. Point-and-Shoot (PaS) / Default
• Scanner is given root URL of the application
• Default configuration remains unchanged
• Minimal human interference
11

12. WAVS
Web application vulnerability scanners are used in two ways:
2. Trained / Configured
• Change configuration (ex. crawl depth)
• Manually visit every page of the application while scanner is in proxy mode.
12
Browser Scanner Proxy Web Application

13. Previous Work
13
• Suto’s case studies:
• 2007 paper evaluated scanners in PaS mode
• 2010 paper evaluated scanners in PaS and Trained modes
• Benchmark applications:
• Web Input Vector Extractor Teaser (WIVET) created in 2009 by Tatli et al.
• Web Application Vulnerability Scanner Evaluation Project (WAVSEP) created in 2010 by
Chen
• Doupé et al.’s 2010 work on evaluating WAVS on the WackoPicko application
• Several other more recent studies evaluate scanners in PaS mode only

14. Methodology

15. Research Goal
• Goal: Performing a comprehensive comparative analysis of the performance of six
chosen scanners in two modes:
• PaS / Default
• Trained / Configured
15
Tool
Selection
Benchmark
Selection
Environment
Setup
Feature &
Metric
Selection
Result
Analysis

16. Tool Selection
• Chen’s evaluation
• Consultation with professional ethical hackers
16
Name Version License Price
Last
Update*
Arachni 1.5.1-0.5.12 Arachni Public Source v1.0 N/A 2017-03-29
Burp Pro 1.7.35 Commercial $349/year 2018-08-29
Skipfish 2.10b Apache v2.0 N/A 2012-12-04
Vega 1.0 MIT N/A 2016-06-29
Wapiti 3.0.1 GNU GPL v2 N/A 2018-05-11
ZAP 2.7.0 Apache v2.0 N/A 2017-11-28
*Checked on August 2018

17. Benchmark Selection
• Benchmark applications:
• WIVET – crawling challenges
• WAVSEP – vulnerability classes
• Intentionally vulnerable realistic web application
• Type of vulnerabilities included in the application
• Architecture of the application and the web technologies used
• Ability of the application to withstand aggressive automated scans
• OWASP Vulnerable Web Applications Directory (VWAD) project
• WackoPicko
17

18. Benchmark Selection - WIVET
• Contains 56 test cases that utilize
both Web 1.0 and Web 2.0
technologies
• Test cases include:
• Standard anchor links
• Links created dynamically using
JavaScript
• Multi-page forms
• Links in comments
• Links embedded in Flash objects
• Links within AJAX requests
18

19. Benchmark Selection - WAVSEP
• Consists of a total of 1220 true positive (TP) test cases and 40 false positive
(FP) test cases
19
Vulnerability Category # of TP test cases # of FP test cases
SQL Injection 138 10
Reflected XSS 89 7
Path Traversal / LFI 816 8
RFI 108 6
Unvalidated Redirect 60 9
DOM XSS 4 0
Passive 5 0

20. Benchmark Selection - WackoPicko
20
• Open-source intentionally vulnerable realistic
web application
• Photo sharing and purchasing site
• Contains 16 vulnerabilities covering several of
the OWASP Top 10
• Contains crawling challenges:
• HTML parsing
• Multi-step process
• Infinite website
• Authentication
• Client-side code

21. Environment Setup 1/2
21
Tools
Applications
*
* VM restored to initial state before every test run

22. Environment Setup 2/2
22
• Each scanner was run in two modes:
• PaS / Default - default configuration setting
• Trained / Configured
1. Maximize crawling coverage – changing
configuration
2. Maximize crawling coverage – use of proxy
3. Maximize attack strength
• WackoPicko test scans were further divided into two
subcategories:
• INITIAL – without authentication / publicly accessible
• CONFIG - valid username/password combination
• In total, each scanner was run eight times

23. Feature and Metric Selection
• Crawling coverage
• % of passed test cases on the WIVET application
• Crawling challenges in the WackoPicko application
• Vulnerability detection accuracy
• TP, FN and FP on the WAVSEP and WackoPicko
applications
• Speed
• Scan time on the WAVSEP and WackoPicko appliations
• Reporting
• Vulnerability detected
• Vulnerability location
• Exploit performed
• Usability
• Efficiency
• Product documentation
• Community support
23
Crawling
Coverage
Detection
Accuracy
Speed
WIVET
WackoPicko
WAVSEP
Features Applications

24. Feature and Metric Selection
• Crawling coverage
• % of passed test cases on the WIVET application
• Crawling challenges in the WackoPicko application
• Vulnerability detection accuracy
• TP, FN and FP on the WAVSEP and WackoPicko
applications
• Speed
• Scan time on the WAVSEP and WackoPicko applications
• Reporting
• Vulnerability detected
• Vulnerability location
• Exploit performed
• Usability
• Efficiency
• Product documentation
• Community support
24
Crawling
Coverage
Detection
Accuracy
Speed
WIVET
WackoPicko
WAVSEP
Features Applications

25. Results

26. Vulnerability Detection Accuracy – FNs 1/4
Vulnerabilities in WackoPicko that were not
detected by any scanners:
1. Weak authentication credentials
• admin/admin
• Reasons:
• Scanners did not attempt to guess
username/password
• Scanners did attempt to guess
username/password but failed
26

27. Vulnerability Detection Accuracy – FNs 2/4
Vulnerabilities in WackoPicko that were not detected
by any scanners:
2. Parameter Manipulation
• Sample user: WackoPicko/users/sample.php?userid=1
Real user: WackoPicko/users/sample.php?userid=2
• Reasons:
• Most scanners did not attempt to
manipulate the userid field
• Arachni manipulated the userid field but
failed to enter a valid number
• Skipfish successfully manipulated the
userid field but did not report it as a
vulnerability 27
userid=2

28. Vulnerability Detection Accuracy – FNs 3/4
Vulnerabilities in WackoPicko that were not detected by any scanners:
3. Sored SQL Injection
4. Directory Traversal
5. Stored XSS
Reasons:
• Crawling challenges – discussed later
• Lack of detection for these types of vulnerabilities
28

29. Vulnerability Detection Accuracy – FNs 4/4
Vulnerabilities in WackoPicko that were not
detected by any scanners:
6. Forceful Browsing
• Access to a link that contains a high quality
version of a picture without authentication
• /WackoPicko/pictures/high_quality.php?key=hig
hquality&picid=11
7. Logic Flaw
• Coupon management functionality
Reasons:
• Require understanding business logic of the
application
• Application specific vulnerabilities
29

30. Vulnerability Detection Accuracy – TPs 1/4
30
WackoPicko Overall Scan Detection Results
Arachni Burp Skipfish Vega Wapiti ZAP
PaS 37.5 37.5 31.25 18.75 25 37.5
Trained 37.5 50 31.25 25 25 43.75
0
10
20
30
40
50
60
70
80
90
100
%ofDetectedVulnerabilities
Key Observations:
• All scanners missed at least 50% of the
vulnerabilities
• In PaS mode Burp, ZAP and Arachni
achieved the same score
• Running the scanners in trained mode
increased the overall detection
• Vega – increase in attack vector
• ZAP & Burp – Manually visiting the pages in
proxy mode for Flash and dynamic JS
technologies

31. 31
WackoPicko Detection Results. The simplest configuration that detected a vulnerability is listed.
Name RXSS XSS
Stored
SQLi
Reflected
Command
line injection
File
Inclusion
File
Exposure
RXSS
behind JS
RXSS
behind
Flash
Arachni INITIAL INITIAL INITIAL INITIAL INITIAL INITIAL
Burp Pro INITIAL INITIAL INITIAL INITIAL INITIAL INITIAL INITIAL CONFIG
Skipfish INITIAL INITIAL INITIAL INITIAL INITIAL
Vega INITIAL INITIAL INITIAL INITIAL
Wapiti INITIAL INITIAL INITIAL INITIAL
ZAP INITIAL INITIAL INITIAL INITIAL INITIAL INITIAL CONFIG
PaS
Trained
• Reminder: INITIAL means w/o authentication credentials and CONFIG means w/ authentication
• Running the scanners in trained mode increased the overall detection
Vulnerability Detection Accuracy – TPs 2/4

32. Vulnerability Detection Accuracy – TPs 3/4
32
WAVSEP Overall TP Detection
Key Observations:
• WAVSEP results were better than
WackoPicko.
• Vulnerability categories in the application
• Integrating WAVSEP in the SDLC of the
scanner
• ZAP achieved highest score, followed by
Vega and Skipfish
Arachni Burp Skipfish Wapiti Vega ZAP
PaS 60.2 27.9 4.0 25.4 71.3 60.7
Trained 60.2 42.5 62.6 24.4 71.3 79.3
0
10
20
30
40
50
60
70
80
90
100
%ofWAVSEPTestsDetected

33. Vulnerability Detection Accuracy – TPs 4/4
33
• Vulnerability category detection varied with scanner
Scanner SQLi RXSS LFI RFI Unvalidated Redirect DOM XSS Passive
Arachni 100% 73% 44% 97% 100% 100% 40%
Burp 97% 83% 24% 60% 76% 0% 0%
Skipfish 78% 80% 64% 33% 36% 0% 40%
Wapiti 50% 60% 15% 44% 0% 100% 0%
Vega 100% 85% 70% 100% 0% 0% 40%
ZAP 86% 74% 74% 100% 100% 0% 40%

34. Crawling Challenges 1/6
Features that scanners found difficult to crawl in
WackoPicko:
1. Uploading a file
• All scanners were not able to upload a
picture in PaS mode
• Burp and ZAP were able to in Trained mode
34

35. Crawling Challenges 2/6
Features that scanners found difficult to crawl in
WackoPicko:
2. Authentication
• All scanners except for Wapiti successfully
created accounts
• None of the scanners used the created
accounts to authenticate
35
Scanner # of Accounts
Arachni 202
Burp 113
Skipfish 364
Vega 117
Wapiti 0
ZAP 111

36. Crawling Challenges 3/6
36
Features that scanners found difficult to
crawl in WackoPicko:
3. Multi-step processes
• All scanners were not able to complete
the process in PaS mode
• Burp and ZAP were able to in Trained
mode

37. Crawling Challenges 4/6
Features that scanners found difficult to crawl in WackoPicko:
4. Infinite websites
• All scanners recognized the infinite loop except Arachni
37
…..
/calendar.php?date=1541454543 /calendar.php?date=1541540943 /calendar.php?date=1541627343

38. Crawling Challenges 5/6
Features that scanners found difficult
to crawl in WackoPicko:
5. Client-side code
• Flash applications
• Dynamic JavaScript
• Ajax Requests
38
Arachni Burp Skipfish Wapiti Vega ZAP
PaS 94 50 50 50 16 42
Trained 94 50 50 50 16 78
0
10
20
30
40
50
60
70
80
90
100
%ofWIVETTestsPassed
WIVET Results

39. Crawling Challenges 6/6
Features that scanners found difficult to crawl in
WackoPicko:
6. State - awareness
• All the scanners exploited SQL injection
vulnerability in login form, however didn’t
discover any of the vulnerabilities that require
authentication
• Vulnerabilities that require authentication
were only discovered in Trained mode
• Credentials given
• Logout link excluded
39
Scanner Web Application

40. Crawling Challenges 6/6
Features that scanners found difficult to crawl in
WackoPicko:
6. State - awareness
• All the scanners exploited SQL injection
vulnerability in login form, however didn’t
discover any of the vulnerabilities that require
authentication
• Vulnerabilities that require authentication
were only discovered in Trained mode
• Credentials given
• Logout link excluded
40
Scanner Web Application

41. Conclusion

42. Conclusion
• Scanners are far from being used as PaS tools only
• Several classes of vulnerabilities were not detected
• Scanners had difficulty crawling through common web architectures
and web technologies
• Different scanners have different strengths/weaknesses
• Open-source scanner performance is comparable to commercial scanner
performance and in several cases better
42

43. Last Words…
To secure a web application you need to find and stop ALL
attack vectors, whereas to break a web application you just
need to exploit ONE attack vector.
43
Web application vulnerability scanners are trying to solve a VERY hard problem!

44. Get in Touch!
44
https://rkhal101.github.io/
/ranakhalil1
@rana__khalil
/rkhal101