1. FFRI,Inc.
Fourteenforty Research Institute, Inc.
FFRI,Inc.
http://www.ffri.jp
Automated on-execute test using VirtualBox
Junichi Murakami
Executive Officer, Director of Advanced Development Division
Ver2.00.01

2. FFRI,Inc.
1. Background and motivation
2. Overview of a test
– automated on-execute test
– virtualization software and automation methods
– Oracle VM VirtualBox and its automation
– example of VBoxManage
3. Automation script
– FFRI AutoMonkey
– design concept
– throughput
– performance
4. References
5. Contact information
Agenda
2

3. FFRI,Inc.
• Automated test against a large amount of malware is required to evaluate a
malware detection engine
• Testing methods are classified into on-demand and on-access testing
• on-execute test which is a kind of on-access test has to execute malware one
by one
• Therefore automation based on virtualization is required
• This slides describes automated on-execute test method using VirtualBox
1.Background and motivation
3
malware scanning on-demand
on-access on-read/write
on-execute
* type of malware scanning

4. FFRI,Inc.
• Basic steps are following
1. Copy malware into a guest
2. Execute copied malware in the guest
3. Analyze or detect malware in the guest
4. Preserve the result after execution is terminated
5. Revert the guest back to original condition
6. Go to 1.
• Required functions to execute above are following
a. Copying a file to a guest from a host (copy-to)
b. Executing arbitrary a program in a guest from a host(exec)
c. Copying a file from guest to a host (copy-from)
d. Reverting a guest condition based on a snapshot(revert)
2.1.Automated on-execute testing
4
→ All functions can be achieved by making a communication interface between a host
and a guest using TCP/IP. We considered the way we do not need to involve
developing software as possible as we could

5. FFRI,Inc.
software Licence copy-to copy-from exec revert method
VMware Workstation Proprietary ○ ○ ○ ○ VIX API
VMware ESX(#1) Proprietary ○ ○ ○ ○ VIX API
Oracle VM VirtualBox GPL2 ○ ○ ○ ○ VBoxManage
QEMU + KVM GPL2(#2) × × × ○ Libvirt
2.2.virtualization software and automation methods
5
#1 ESXi can also use VIX API for 60days by registering a evaluation license.
#2 KVM's parts are licensed under various GNU licenses(GPL, GPL2, LGPL2, etc.)
• Use functions which virtualization software has natively
• VMware(licensed) and VritualBox have all the features we need
→ We considered using VirtualBox because of the cost advantage
• QEMU+KVM can be used by 3rd party software(ex: libguestfs + winexe)
– “Malware Analysis: Collaboration, Automation & Tuning”, Shmoocon 2013
http://www.slideshare.net/xabean/malware-analysis-16674048

6. FFRI,Inc.
• A kind of x86 virtualization software, currently developed by Oracle
• Version 4.0 and later, fully open source software (GPL2)
• Supporting various host and guest environments
– HostOS：Windows, Linux, Mac OS X, Solaris
– GuestOS：Windows, Linux, FreeBSD, OpenBSD, Mac OS X Server, Solaris,etc.
• CLI is available (VBoxManage), friendly to automation
– startvm , pause, resume, poweroff, clonevm, showvinfo
– copyto, copyfrom, exec
– taking snapshot and reverting
– control virtual machine devices status, etc.
2.3.Oracle VM VirtualBox and its automation
6

7. FFRI,Inc.
2.4.Example of VBoxManage
7
% vboxmanage startvm vm
% vboxmanage controlvm vm poweroff
% vboxmanage snapshot vm restore snapshot-1
% vboxmanage guestcontrol exec vm --image “c:/windows/system32/calc.exe” ¥
--username admin --timeout 60000 --wait-exit
% vboxmanage guestcontrol vm copyto “/some/file” “c:/file.txt” --username admin
* starting a guest
* power off a guest
* reverting a guest based on a snapshot
* execute a program in a guest from a host
* copying a file to a guest from a host

8. FFRI,Inc.
• Automation script using VBoxManage, just a shell script
– auto-monkey.sh:automation for copy, exec, copy, revert steps
– watch-monkey.sh：watch dog script for the monkey
• It can execute multiple test simultaneously, works individually
• Published at our website below, see README for the detail (License: BSD)
– http://www.ffri.jp/research/freeware.htm
3.1.FFRI AutoMonkey
8
VirtualBox(guest)
malware
log
auto-monkey.sh
watch-monkey.sh
VirtualBox(guest)
malware
log
auto-monkey.sh
watch-monkey.sh
image
snapshot
image
snapshot

9. FFRI,Inc.
• conform to KISS principle
• Estimation of remaining time is important for this kind of test
– we cannot determine when it would finish if the script hangs up
• Stability of VBoxManage (and VIX API) is the lifeline for the automation
• In fact, error occurs when it runs long time
– Failure by error
• exits immediately
• resumed a test automatically by watch-monkey.sh
– Hanging up(stuck) by error
• watch-monkey.sh monitors lifetime of a VirtualBox process
• if it is stuck, kill and resume
3.2.Design concept
9

10. FFRI,Inc.
• Testing under 1host and 7guest environment
• Processed 20,000 malware, each execution time was 60 seconds
– total elapsed time: 37h15m
– throughput：8.95 malware/minute
# if malware execution terminated less than 60 seconds, the script processes next item.
• Host and guest environment is following
3.3.Throughput
10
Hardware CPU: Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz
Memory: 8GB
HDD: 1.8TB x 1
Host OS Ubuntu 13.04 + VirtualBox 4.2
Guest OS Windows XP SP3(x86) + FFR yarai 2.3
CPU:1 CPU
Memory:750MB

11. FFRI,Inc.
• About 70% of total processor is idle state(each core also indicates same trend)
3.4.performance - processor
11
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0:05
0:55
1:45
2:35
3:25
4:15
5:05
5:55
6:45
7:35
8:25
9:15
10:05
10:55
11:45
12:35
13:25
14:15
15:05
15:55
16:45
17:35
18:25
19:15
20:05
20:55
21:45
22:35
23:25
load
all processsor statistics
%idle
%steal
%iowait
%system
%nice
%user

12. FFRI,Inc.
• Consuming about 80% - 90% memory steadily
– real memory usage is between 2.5 and 4.0GB
3.4.performance - memory
12
0
500000
1000000
1500000
2000000
2500000
3000000
3500000
4000000
4500000
5000000
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0:05
1:05
2:05
3:05
4:05
5:05
6:05
7:05
8:05
9:05
10:05
11:05
12:05
13:05
14:05
15:05
16:05
17:05
18:05
19:05
20:05
21:05
22:05
23:05
memory utilization statistics
kbmemused
kbmemfree
kbmemused
-(kbbuffers+kbcached)

13. FFRI,Inc.
• Disk busy ratio(%util) stays around 30% steadily
• The number of queued requests is between 4 and 8
3.4.performance – Disk IO
13
0
10
20
30
40
50
60
70
80
90
0
2
4
6
8
10
12
0:05
0:55
1:45
2:35
3:25
4:15
5:05
5:55
6:45
7:35
8:25
9:15
10:05
10:55
11:45
12:35
13:25
14:15
15:05
15:55
16:45
17:35
18:25
19:15
20:05
20:55
21:45
22:35
23:25
0:00
Usage of block device(/dev/sda)
avgqu-sz
%util

14. FFRI,Inc.
• None of CPU, memory and IO wasn’t bottleneck under 1host and 7guest
environment
• It seems we can add some more guests up to around 10 VMs according to
memory usage
• However, we have to consider requirement of a process which is executed in
a guest (cpu, memory)
3.4.performance - consideration
14

15. FFRI,Inc.
• http://www.ffri.jp/assets/files/research/freeware/FFRIAutoMonkey-1.0.tgz
• https://www.virtualbox.org/manual/UserManual.html
• http://www.slideshare.net/xabean/malware-analysis-16674048
• http://www.youtube.com/watch?v=peHdyUlchSM
• http://libguestfs.org/
• http://sourceforge.net/projects/winexe/files/
References
15

16. FFRI,Inc.
• E-Mail
– research-feedback@ffri.jp
• Twitter
– @FFRI_Research
Contact Information
16