AsiaBSDCon 2012 Tokyo University of Science, Tokyo, Japan 22-25 March, 2012 Mar 25 17:30 - 18:30 Work-In-Progress session

1. Running FreeBSD
on Linux KVM Virtualization
©2003 Project TIMEMACHINE
AsiaBSDCon 2012 Tokyo
March 25, 2012
Takeshi HASEGAWA
[ja] @hasegaw / [en] @hasegaw_en

2. Takeshi Hasegawa
Interests
• Virtualization (Xen, and KVM)
• Storage
• Snowboarding J In this winter I had great days with my colleges
• Poor at coding. I’m not a software developer.
Work Experience
• April 2004- Systems Engineer at Software Research Associates, Inc.
• June 2011- Sales Engineer at Fusion-io
2

3. My works – books and magazines I participated
The Best-seller UNIX admin guide with catoon
Best-seller Xen book in Japan.
characters. More than 15k copies are sold
More than 6k copies are sold
Our sponsor, Gihyo Books!
I discussed the benfit of
Linux KVM.
And more…
3

4. Running articles on the Web
エンジニアなら知っておきたい仮想マシンのしくみ
(Now you understand how virtual machine works)
http://gihyo.jp/dev/serial/01/vm_work/ 4

5. BSD and myself
• Met FreeBSD 2.2.5 in 1997
• Open Source no matsuri 2000
o Met JKH and Warner Losh
• 2001~2006 This picture is still my treasure! J
o Contracted a dedicated Server with FreeBSD (4.x)
• 2010
o Used V-IMAGE for the network basic session of new-hire training
o Wrote virtio drivers for FreeBSD
• 2011-2012
o Wrote KVM PVClock for FreeBSD
IP Networking amiation DEMO
Powered by VIMAGE and HTML5
5

6. Running FreeBSD
on Linux KVM Virtualization
6

7. Why FreeBSD on Linux KVM?
More opportunities to run FreeBSD
• Many service providers are running KVM
o VPS (Virtual Private Server)
o Cloud – such as Sakura Internet (#sacloud)
• Linux KVM is surely handy
o The free-of-charge Type-1 hypervisor
o Guest OS doesn’t require device drivers for the latest hardware
Co-existence of FreeBSD allows leverage of its specialties
• ZFS, BSD Network stacks, jail/vimage, …
• and more!
7

8. However – FreeBSD on Linux KVM runs slow
The biggest Bottlenecks seen
in Virtual Machines is: The solution is:
• I/O Bottlenecks • I/O Para-virtualization
– Disk I/O – Disk para-virtualization
– Network I/O – Network para-virtualization
seq$write$MB/s$
90"
80"
70"
60"
50"
40" seq"write"MB/s"
30"
20"
10"
0"
Na.ve" FreeBSD"VM"+"SCSI"Emu"
Especially FreeBSD is known as
“low-performance guest ” on Linux KVM.
8

9. virtio – I/O interface for virtualized guests
• Ring-Buffer Interface definition
between Guest and Host
Guest
• Independent to hypervisors
– lguest
– Linux KVM
virtio virtio virtio virtio
– VirtualBox Ballon Block Block Net
– BHyVe
– Xen
last summer, personally I was also working the
virtio support on Xen.... virtio virtio virtio virtio
Ballon Block Block Net
PCI Bus
• Works efficiently Host
– No overhead of H/W emulation
– Less Intel VT/AMD-V world
changes
– Less context switches Hardware
9

10. virtio drivers for FreeBSD
• I wrote virtio drivers for FreeBSD – Fall of 2010
o Rusty Russel, Father of virtio, made a presentation at LinuxCon 2010
“virtio-net guest driver can be implemented by less than 1,000 lines”
o I actually tried on FreeBSD 8.1-R. It took more than 1,000 lines! J
o Finally, I ported:
• virtio interface
• virtio-net (network)
• virtio-blk (block)
• virtio driver for FreeBSD merged to -CURRENT – Winter of 2011
o Congrats!
o Please note - the merged one is a DIFFERENT work from mine!
10

11. virtio-net increases network performance
• 4.7x against rtl8139, 1.3x against e1000 bandwidth improvement!
1.3x
4.7x
Test Environment: HP ML115G1, 1.8G 1P2C, PC6400 DRAM (too poor!)
11
Fedora 14 as the Host, and running FreeBSD 8.1-RELEASE as the Guest(2vCPUs)

12. virtio-blk increases block I/O performance …
seq$write$MB/s$
90"
80"
70"
60"
50"
40" seq"write"MB/s"
30"
20"
10"
0"
Na.ve" FreeBSD"VM"+"SCSI"Emu"
12

13. virtio-blk increases block I/O performance a bit
seq$write$MB/s$
90"
80"
70"
60"
50"
40" seq"write"MB/s"
30" OOPS!
20"
10"
0"
Na.ve" FreeBSD"VM"+"SCSI" FreeBSD"VM"+"
Emu" vir.o"
13

14. Hey… what’s happening?
14

15. “Okay I’ll look into that…”
15

16. Latency Distribution – CentOS Guest w/virtio
Distribution(%)
91"
Linux guest performs native-level
performance. Ideal!
81"
>79% of IOs are done in <10ms
71"
61" CentOS"5"host"
51"
41"
CentOS"VM+Vir;o"
31"
21"
11"
1"
0" 0.5" 1" 1.5" 2" 2.5" 3" 3.5" 4" 4.5"
Low latency High
16

17. Latency Distribution – FreeBSD Guest
Only 38% of IOs are done in <10ms
Distribution(%)
91"
81" FreeBSD guest has some latency
71" bottleneck…
61" FreeBSD"VM+Vir6o"
51"
Fedora"14"HDD"
41"
FreeBSD"VM+SCSI"Emu"
31"
21"
11"
1"
0" 1" 2" 3" 4" 5" 6"
Low latency High
17

18. Latency Distribution –
FreeBSD/CentOS Comparision
Distribution(%)
91"
81"
Fedora"14"HDD"
71"
61"
CentOS5"VM"
51"
41"
FreeBSD"VM+Vir>o"
31"
21"
FreeBSD"VM+SCSI"
11" Emu"
1"
0" 1" 2" 3" 4" 5" 6"
Low latency High
18

19. I/O Characteristics between FreeBSD n’ Linux
fio
--filename=/dev/(virtio
device)
--direct=1
--rw=randwrite
--bs=4k
--size=2G
--numjobs=1
--runtime=10
--group_reporting
--name=a
CentOS 5.7 Guest on CentOS 5.7
Device:
rrqm/s
wrqm/s
r/s
w/s
rMB/s
wMB/s
avgrq-sz
avgqu-sz
await
svctm
%util
vdb
0.00
0.00
0.00
162.00
0.00
0.63
8.00
0.97
5.98
5.98
96.90
vdb
0.00
0.00
0.00
151.49
0.00
0.59
8.00
0.96
6.29
6.31
95.64
vdb
0.00
0.00
0.00
161.00
0.00
0.63
8.00
0.97
6.09
6.04
97.30
vdb
0.00
0.00
0.00
155.00
0.00
0.61
8.00
0.97
6.20
6.26
97.00
vdb
0.00
0.00
0.00
159.00
0.00
0.62
8.00
0.98
6.23
6.16
98.00
FreeBSD 8.1 Guest on Fedora 14
Device:
rrqm/s
wrqm/s
r/s
w/s
rMB/s
wMB/s
avgrq-sz
avgqu-sz
await
svctm
%util
dm-6
0.00
0.00
0.00
178.00
0.00
0.35
4.00
0.70
3.88
3.93
70.00
dm-6
0.00
0.00
0.00
160.00
0.00
0.31
4.00
0.73
4.64
4.58
73.30
dm-6
0.00
0.00
0.00
172.00
0.00
0.34
4.00
0.68
3.94
3.98
68.40
dm-6
0.00
0.00
0.00
164.00
0.00
0.32
4.00
0.69
4.20
4.21
69.00
dm-6
0.00
0.00
0.00
168.00
0.00
0.33
4.00
0.75
4.51
4.47
75.10
1. IO reqs from FreeBSD is intermittent, and the disk is sometimes idle
2. It seems that Linux queues 2x IO reqs/queue more than FreeBSD 19

20. iostat results in proliant2 (Fedora14)
Device:
rrqm/s
wrqm/s
r/s
w/s
rMB/s
wMB/s
avgrq-sz
avgqu-sz
await
svctm
%util
FreeBSD+SCSI
Emu
dm-6
0.00
0.00
0.00
260.00
0.00
0.51
4.00
0.94
3.63
3.63
94.30
dm-6
0.00
0.00
0.00
264.00
0.00
0.52
4.00
0.94
3.54
3.55
93.80
dm-6
0.00
0.00
0.00
270.00
0.00
0.53
4.00
0.97
3.62
3.60
97.20
dm-6
0.00
0.00
0.00
270.00
0.00
0.53
4.00
0.97
3.58
3.59
96.90
dm-6
0.00
0.00
0.00
264.00
0.00
0.52
4.00
0.97
3.66
3.69
97.40
Device:
rrqm/s
wrqm/s
r/s
w/s
rMB/s
wMB/s
avgrq-sz
avgqu-sz
await
svctm
%util
FreeBSD+VirtIO
dm-5
0.00
0.00
0.00
266.00
0.00
0.52
4.00
0.99
3.74
3.73
99.10
dm-5
0.00
0.00
0.00
278.00
0.00
0.54
3.97
0.99
3.55
3.56
99.00
dm-5
0.00
0.00
0.00
296.00
0.00
0.58
4.03
1.02
3.46
3.29
97.30
dm-5
0.00
0.00
0.00
272.00
0.00
0.53
4.00
1.03
3.79
3.62
98.40
dm-5
0.00
0.00
0.00
260.00
0.00
0.51
4.00
0.99
3.82
3.81
99.10
Device:
rrqm/s
wrqm/s
r/s
w/s
rMB/s
wMB/s
avgrq-sz
avgqu-sz
await
svctm
%util
Linux
guest
vdb
0.00
0.00
0.00
154.46
0.00
0.60
8.00
3.94
24.20
6.42
99.21
vdb
0.00
0.00
0.00
146.00
0.00
0.57
8.00
3.98
28.44
6.86
100.20
vdb
0.00
0.00
0.00
147.00
0.00
0.57
8.00
3.98
26.01
6.81
100.10
vdb
0.00
0.00
0.00
140.00
0.00
0.55
8.00
4.00
29.05
7.16
100.30
vdb
0.00
0.00
0.00
159.00
0.00
0.62
8.00
3.98
24.55
6.30
100.20
vdb
0.00
0.00
0.00
165.00
0.00
0.64
8.00
3.99
24.22
6.07
100.10
Device:
rrqm/s
wrqm/s
r/s
w/s
rMB/s
wMB/s
avgrq-sz
avgqu-sz
await
svctm
%util
Linux
native
dm-5
0.00
0.00
0.00
259.00
0.00
1.01
8.00
3.97
15.27
3.86
100.00
dm-5
0.00
0.00
0.00
259.00
0.00
1.01
8.00
3.97
15.37
3.86
100.00
dm-5
0.00
0.00
0.00
258.00
0.00
1.01
8.00
3.97
15.50
3.88
100.00
dm-5
0.00
0.00
0.00
250.00
0.00
0.98
8.00
3.99
15.88
4.00
100.00
dm-5
0.00
0.00
0.00
165.00
0.00
0.64
8.00
3.98
16.48
6.06
100.00
20

21. virtio-blk can make FreeBSD Guests faster?
No!!
Facts
• I/O latency from virtual machine is more than twice of native I/O
• The performance issue is seen in block layer
• virtio-blk surely improved latency distribution (compared to SCSI
emu), but it is still ignorable impact
Then… what is the bottleneck in the FreeBSD Guests??
o Interrupt delivery?
o Queueing I/Os to the block device?
More deep investigation should be done J
21

22. My another work – KVM Para-virtualized Clock
Motivation
• I wanted more reliable timer needed to investigate performace issue
of FreeBSD on Linux KVM
What’s KVM PVClock?
• clocksource which can be found in Linux KVM environment
o “Timecounter” source in FreeBSD world
o More accurate VM time-keeping can be expected
• Inspired by Xen
o Shadows TSC counter of the host processor, and calendar of the host.
o The replica of timer virtualization in Xen
o FreeBSD already has Xen clock support
22

23. The result of the early implementaiton
while true; do date; sleep 1; done
on FreeBSD Guest
The time-slip is less than one second
- The two OSes should be in synchronized
theoretically
- No any time-synchronization
method required
- tested for 120hours (5 days)
while true; do date; sleep 1; done
on Linux KVM Host
23

24. Conclusion
• Linux KVM will bring more opportunities to run FreeBSD
• Linux KVM uses virtio for I/O para-virtualizaton
• FreeBSD block layer does not fit well with Linux KVM
• Porting Linux KVM PVClock for FreeBSD
• Any comments? Questions? Suggestions?
• please say hello to me after the session.
24

25. Thank you!
25