Open Source Virtualization based on KVM and available tools like Libvirt, Libguestfs, Qemu, ...

1. Virtualize with KVM
Tools and Enterprise usage
by Vincent Van der Kussen

2. #whoami
● Using Linux since 1996
● Sysadmin / Engineer / Opensource Enthousiast
● @vincentvdk
● http://blog.vanderkussen.org

3. So what is this KVM thing anyway?

4. It's NOT!

7. KVM : overview
Kernel-based Virtual Machine, a Kernel module that
turns Linux into a hypervisor
Created by Quramnet (bought by Red Hat 2008)
Supports x86 (32 and 64 bit), s390, Powerpc
Included in Linux kernel since 2.6.20
Needs a CPU with virtualization extention !

8. KVM : overview
● Each vm is a process
● Each virtual cpu is a thread
● Open source

9. Features
CPU and memory overcommit
High performance paravirtual I/O
Hotplug (cpu, block, nic)
SMP guests
Live Migration
Power management

10. Features
PCI Device Assigenment and SR-IOV
KSM (Kernel Samepage Merging)
SPICE
NUMA

11. KVM : differences
Not your typical VMWare workstation or VirtualBox.
VirtManager comes close but still a little rough on the
edges
KVM is meant for virtualization on servers. Desktop
usage is not always ideal
(think vSphere, HyperV,..).

12. Overview Tools
libvirt (api)
Virsh
Qemu
VirtManager
Libguestfs
virt-v2v

13. Overview tools
sVirt
virt-*
Boxgrinder
VirtIO (not really a tool)
Spice

14. Libvirt : The Virtualization API

15. Libvirt : overview
● “The Virtualization API”
● Toolkit to interact with the virtualization layer
● Written in C
● LGPL
● Has bindings for common languages (Python,
Ruby, Perl, PHP, Java,..)
● Supports AMQP via libvirt-qpid

16. Libvirt : overview
● Provides management of
– Virtual machines
– Virtual networks
– Storage
● Does NOT provide multi node management
features like load balancing
● Suspend / resume support

17. Libvirt : terminology
Node
a physical machine

18. Libvirt : termonology
Hypervisor
A software layer that allows a node to run virtual
machines

19. Libvirt : terminologie
Domain
An instance (“virtual machine”) running on the
hypervisor

21. Libvirt : overview
Supports also :
VMWare
Microsoft HyperV
XEN
KVM
OpenVZ
Virtualbox LXC

22. virsh

23. Virsh overview
Command line for libvirt
Enables scripting
Libvirtd needs to be started
Most commands require root privileges

24. Virsh usage
virsh <command> <domain-id> [OPTIONS]
virsh <command> <domain-id> [OPTIONS]

25. Virsh usage
Connect to hypervisor on the localhost
virsh connect qemu:///system
virsh connect qemu:///system
or
virsh
virsh
Connect to hypervisor on remote system over SSH
virsh -c qemu+ssh://user@host.domain.foo/system
virsh -c qemu+ssh://user@host.domain.foo/system

26. Virsh usage

27. Virsh : storage
● Define storage pool
● Local filesystem (ext3, ext4,...)
● Directory based
● ISCSI
● NFS
● LVM (uses volume group)
● Physical disk
● multipath

28. Virsh : create storage pool
<pool type="dir">
<name>LOCAL-STORAGE</name>
Xml file -> <target>
<path>/VMS</path>
</target>
</pool>
[root@localhost ~]# virsh pool-define dir_pool.xml
[root@localhost ~]# virsh pool-define dir_pool.xml
Pool LOCAL-STORAGE defined from dir_pool.xml
Pool LOCAL-STORAGE defined from dir_pool.xml
[root@localhost ~]# virsh pool-start LOCAL-STORAGE
[root@localhost ~]# virsh pool-start LOCAL-STORAGE
Pool LOCAL-STORAGE started
Pool LOCAL-STORAGE started

29. Virsh : create storage pool
[root@localhost ~]# virsh pool-list
[root@localhost ~]# virsh pool-list
Name
Name State
State Autostart
Autostart
-----------------------------------------
-----------------------------------------
default
default active
active yes
yes
LOCAL-STORAGE
LOCAL-STORAGE active
active no
no
[root@localhost ~]# virsh
[root@localhost ~]# virsh pool-autostart LOCAL-STORAGE
pool-autostart LOCAL-STORAGE
Pool LOCAL-STORAGE
Pool LOCAL-STORAGE marked as autostarted
marked as autostarted

30. Virsh : create VM
● VM is defined in xml file
● Manual creation of VM's disks
[root@localhost ~]# qemu-img create
[root@localhost ~]# qemu-img create /VMS/vm02-fromxml.img 5G
/VMS/vm02-fromxml.img 5G
Formatting '/VMS/vm02-fromxml.img',
Formatting '/VMS/vm02-fromxml.img', fmt=raw size=5368709120
fmt=raw size=5368709120
● Define VM in libvirt and start it
[root@localhost ~]# virsh define vm02_from_xml.xml
[root@localhost ~]# virsh define vm02_from_xml.xml
Domain vm02-fromxml defined from vm02_from_xml.xml
Domain vm02-fromxml defined from vm02_from_xml.xml
[root@localhost ~]# virsh start vm02-fromxml
[root@localhost ~]# virsh start vm02-fromxml
Domain vm02-fromxml started
Domain vm02-fromxml started

31. 1 <domain type='kvm'>
2 <name>vm02-fromxml</name>
3 <uuid></uuid>
4 <memory>128288</memory>
5 <currentMemory></currentMemory>
6 <vcpu>2</vcpu>
7 <os>
8 <type arch='x86_64' machine='pc-0.14'>hvm</type>
9 <boot dev='hd'/>
10 </os>
11 <features>
12 # features such as ACPI are defined here
13 </features>
14 <devices>
15 <emulator>/usr/bin/qemu-kvm</emulator>
16 <disk type='file' device='disk'>
17 <driver name='qemu' type='raw' cache='none'/>
18 <source file='/VMS/vm02-fromxml.img'/>
19 <target dev='vda' bus='virtio'/>
20 <alias name='virtio-disk0'/>
21 <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
22 </disk>
23 </devices>
24 </domain>

32. Virsh : list vm's
[root@localhost ~]# virsh list --all
[root@localhost ~]# virsh list --all
Id Name
Id Name State
State
----------------------------------
----------------------------------
4 vm01
4 vm01 running
running
- vm02-fromxml
- vm02-fromxml shut off
shut off
[root@localhost ~]#
[root@localhost ~]#

33. Virsh : live migration
virsh migrate --live <guestname> qemu+ssh://<destination host>/system
virsh migrate --live <guestname> qemu+ssh://<destination host>/system

34. Virsh : snapshots
● Only on qcow2 images
# virsh snapshot-create <domain>
# virsh snapshot-create <domain>
# virsh snapshot-list <domain>
# virsh snapshot-list <domain>
Name
Name Creation Time
Creation Time State
State
---------------------------------------------------
---------------------------------------------------
1295973577
1295973577 2011-01-25 17:39:37 +0100 running
2011-01-25 17:39:37 +0100 running
1295978837
1295978837 2011-01-25 19:07:17 +0100 running
2011-01-25 19:07:17 +0100 running
# virsh snapshot-restore <domain> <snapshotname>
# virsh snapshot-restore <domain> <snapshotname>

35. Qemu

36. qemu
● KVM istself cannot create VM's
● KVM is “just” a hypervisor
● Qemu as user space process
● Qemu communicates via /dev/kvm

37. qemu
● Qemu-img
– Check
– Convert
– Create
– Snapshot
– Info

38. Qemu
Inspect image
[root@HV01 ~]# qemu-img info /var/lib/libvirt/images/WIN7.img
[root@HV01 ~]# qemu-img info /var/lib/libvirt/images/WIN7.img
image: /var/lib/libvirt/images/WIN7.img
image: /var/lib/libvirt/images/WIN7.img
file format: raw
file format: raw
virtual size: 12G (12884901888 bytes)
virtual size: 12G (12884901888 bytes)
disk size: 12G
disk size: 12G
[root@HV01 ~]#
[root@HV01 ~]#

39. qemu
Example :
Convert raw to vmdk image
qemu-img convert -O vmdk <raw image> outimage.vmdk
qemu-img convert -O vmdk <raw image> outimage.vmdk

40. Qemu
Supported disk formats
● file ● cow
● blkverify ● Raw
● sheepdog ● vmdk
● blkdebug ● vdi
● qcow2 ● qcow
● vvfat

41. virtio
● "full virtualization" is a nice feature because It
allows you to run any operating system
virtualized.
● However, it's slow because the hypervisor has
to emulate actual physical devices.

42. virtio
● Virtio drivers solve this
● Virtio drivers know they run in a virtualized
environment. (no emulation)
● Virtio drivers operate with hypervisor

43. virt-*

44. virt-*
● Virt-install
● Virt-viewer
● Virt-clone
● virt-top

45. Virt-install
● Can create VM and disks in one go
[root@localhost ~]# virt-install --connect qemu:///system --graphics
[root@localhost ~]# virt-install --connect qemu:///system --graphics
vnc,listen=0.0.0.0 --name=vm01 --ram 512 --disk pool=LOCAL-
vnc,listen=0.0.0.0 --name=vm01 --ram 512 --disk pool=LOCAL-
STORAGE,size=10,bus=virtio,cache=none /
STORAGE,size=10,bus=virtio,cache=none /
--network network=default,model=virtio /
--network network=default,model=virtio /
--location=http://ftp.debian.org/debian/dists/squeeze/main/installer-amd64
--location=http://ftp.debian.org/debian/dists/squeeze/main/installer-amd64
Cannot open display:
Cannot open display:
Run 'virt-viewer --help' to see a full list of available
Run 'virt-viewer --help' to see a full list of available command line
command line
options
options
Domain installation still in progress. You can reconnect
Domain installation still in progress. You can reconnect to
to
the console to complete the installation process.
the console to complete the installation process.

46. Virt-viewer
● Connect to the VM's console with
– RDP
– VNC
– SPICE
● Default vnc port -> 5900
● virt-viewer
virt-viewer -c qemu:///system xpsp2
virt-viewer -c qemu:///system xpsp2

47. virt-clone
● Clones an existing VM (Duh!!)
● Copies disk images
● Defines new machine based on source
– MAC address is auto generated
● Deploy many similar machines from one image
(template).

48. virt-clone
[root@HV01 ~]# virt-clone -o vm01 -n vm01-copy -f /
[root@HV01 ~]# virt-clone -o vm01 -n vm01-copy -f /
/VMS/vm01-copy.img
/VMS/vm01-copy.img
Allocating 'vm01-copy.img'
Allocating 'vm01-copy.img' | 10 GB
| 10 GB 03:37
03:37
Clone 'vm01-copy' created successfully.
Clone 'vm01-copy' created successfully.
[root@HV01 ~]#
[root@HV01 ~]#

49. virt-top
virt-top 20:40:19 - x86_64 2/2CPU 2666MHz 3962MB
virt-top 20:40:19 - x86_64 2/2CPU 2666MHz 3962MB
4 domains, 4 active, 4 running, 0 sleeping, 0 paused, 0 inactive D:0 O:0 X:0
4 domains, 4 active, 4 running, 0 sleeping, 0 paused, 0 inactive D:0 O:0 X:0
CPU: 2.8% Mem: 2560 MB (2560 MB by guests)
CPU: 2.8% Mem: 2560 MB (2560 MB by guests)
ID S RDRQ WRRQ RXBY TXBY %CPU %MEM
ID S RDRQ WRRQ RXBY TXBY %CPU %MEM TIME
TIME NAME
NAME
14
14 R
R 0
0 0 1220
0 1220 0
0 2.5
2.5 25.0 315:52.95 WIN7
25.0 315:52.95 WIN7
15
15 R
R 0
0 4 146
4 146 42
42 0.2
0.2 12.0
12.0 0:54.64 vm01
0:54.64 vm01
11
11 R
R 0
0 0
0 0.2
0.2 12.0 15:38.34 vm02-fromxml
12.0 15:38.34 vm02-fromxml
16
16 R
R 0
0 0 104
0 104 0
0 0.0
0.0 12.0
12.0 0:54.82 vm01-copy
0:54.82 vm01-copy

50. Virtual machine manager

51. Virtual Machine Manager
● A GUI for virsh and virt-*
● Runs only on Linux/Unix
● Can discover machines running libvirtd (nodes)
● Local development / testing

52. Virtual Machine Manager

53. libguestfs
libguestfs :
Converting, Inspecting and modifying VM's

54. libguestfs
● Manipulation of VM images
● Inspection of VM images
● Exists out of several separate tools
– List is available on http://virt-tools.org/docs/index.html
● No need to login on the VM itself
● Can be run on an active VM

55. What is this image?
Is it still needed?
Developers like to keep stuff

56. Libguestfs : virt-inspector
● What if you just have an image
● You cannot login to machine
● You would like to know what version of a
package is installed
● Generate list of all VM's

57. Libguestfs : virt-inspector
Inspect the domain
virt-inspector -d WIN7
virt-inspector -d WIN7
Inspect an image
[root@HV01 VMS]# virt-inspector -a /var/lib/libvirt/images/WIN7.img
[root@HV01 VMS]# virt-inspector -a /var/lib/libvirt/images/WIN7.img

58. 1 <name>windows</name>
2 <arch>i386</arch>
3 <distro>windows</distro>
4 <product_name>Windows 7 Professional</product_name>
5 <product_variant>Client</product_variant>
6 <major_version>6</major_version>
7 <minor_version>1</minor_version>
8 <windows_systemroot>/Windows</windows_systemroot>
9 <windows_current_control_set>ControlSet001</windows_current_control_set>
10 <hostname>btr-PC</hostname>
11 <format>installed</format>

59. 1 <applications>
2 <application>
3 <name>Mozilla Firefox 6.0.2 (x86 en-US)</name>
4 <display_name>Mozilla Firefox 6.0.2 (x86 en-US)</display_name>
5 <version>6.0.2</version>
6 <install_path>C:Program FilesMozilla Firefox</install_path>
7 <publisher>Mozilla</publisher>
8 <url>http://www.mozilla.com/en-US/</url>
9 <description>Mozilla Firefox 6.0.2 (x86 en-US)</description>
10 </application>
11 </applications>

60. Libguestfs : virt-df
● Allows you to view filesystem layout and space free/used
Virt-df <name of domain>
[root@HV01 ~]# virt-df vm02-fromxml
[root@HV01 ~]# virt-df vm02-fromxml
Filesystem
Filesystem 1K-blocks
1K-blocks Used
Used Available
Available Use%
Use%
vm02-fromxml:/dev/sdb
vm02-fromxml:/dev/sdb 3516418
3516418 3516418
3516418 0
0 100%
100%
vm02-fromxml:/dev/sda1
vm02-fromxml:/dev/sda1 495844
495844 30573
30573 439671
439671 7%
7%
vm02-fromxml:/dev/VolGroup/lv_root
vm02-fromxml:/dev/VolGroup/lv_root 3652680
3652680 623180
623180 2843952
2843952 18%
18%
[root@HV01 ~]#
[root@HV01 ~]#

61. Libguestfs : virt-win-reg
● Modify registry of Windows machines
virt-win-reg --merge <vmname>.img viostor-reg
virt-win-reg --merge <vmname>.img viostor-reg

62. Libguestfs : virt-win-reg
Manually Prepare a Windows image with virtio drivers
virt-win-reg --merge <vmname>.img viostor-reg
virt-win-reg --merge <vmname>.img viostor-reg
guestfish -i <vmname>.img
guestfish -i <vmname>.img
<fs>upload viostor.sys
<fs>upload viostor.sys
/WINDOWS/system32/drivers/viostor.sys
/WINDOWS/system32/drivers/viostor.sys

63. Libguestfs : virt-tar-out
● Create backups from files on the VM
# virt-tar-out -d vm01 /home home-vm01.tar
# virt-tar-out -d vm01 /home home-vm01.tar
● Create backups from files on the disk image
# virt-tar-out -a /VMS/vm01.img /home home-vm01-disk.tar
# virt-tar-out -a /VMS/vm01.img /home home-vm01-disk.tar

64. virt-v2v :
migration of virtual machines

65. Virt-v2v : short overview
● Tool to automate migrations to KVM/RHEV
● Migrate to KVM or RHEV from
– XEN
– VMWare
– KVM
● Installs virtio drivers and reconfigures machine
● Can be scripted for bulk operations

66. Virt-v2v : migrate from Vmware to
RHEV

67. Virt-v2v : migrate from VMWare to
RHEV
virt-v2v -ic esx://esx.example.com/?no_verify=1
virt-v2v -ic esx://esx.example.com/?no_verify=1 -o rhev
-o rhev
-os storage.example.com:/exportdomain --network
-os storage.example.com:/exportdomain --network rhevm vm-
rhevm vm-
name
name

68. Virt-v2v : migrate VMWare to KVM

69. sVirt : secure your host and VM's

70. sVirt
Virtualization introduces new risks

71. sVirt
“Old” days
Machines where physically separated
and “attackable” via the network

73. sVirt
With Virtualization :
A flawed hypervisor can give access to ALL VM's
running on it
Bypassing network security

75. sVirt
sVirt :
Apply MAC for guest and resources
MAC policy is set by hypervisor host
Based on SELinux
(don't reinvent the wheel)
Auto apply or manual

77. Disk images
[root@HV01 VMS]# ls -Z |grep vm01
[root@HV01 VMS]# ls -Z |grep vm01
qemu qemu system_u:object_r:svirt_image_t:s0:c656,c817
qemu qemu system_u:object_r:svirt_image_t:s0:c656,c817 vm01-copy.img
vm01-copy.img
qemu qemu system_u:object_r:svirt_image_t:s0:c690,c972
qemu qemu system_u:object_r:svirt_image_t:s0:c690,c972 vm01.img
vm01.img
Processes
[root@HV01 VMS]# ps auxZ | grep vm01 |awk '{print $1"
[root@HV01 VMS]# ps auxZ | grep vm01 |awk '{print $1" " $22}'
" $22}'
system_u:system_r:svirt_t:s0:c690,c972
system_u:system_r:svirt_t:s0:c690,c972 vm01
vm01
system_u:system_r:svirt_t:s0:c656,c817
system_u:system_r:svirt_t:s0:c656,c817 vm01-copy
vm01-copy

78. A lot of tools but what to do with them?

79. RHEV is first example

80. Build your own stack?
Development environment?

81. Enterprise usage

82. RHEV 2.x
● RHEV (Red Hat enterprise Virtualization)
– Red Hat started adopting KVM (actually they bought it)
– Released 2.1
– Running on Windows platform ( the horror.. )
– RHEV 2.2 will be latest release in 2.x series

83. RHEV 3.0
● Port from .Net to Java
● Manager runs on Jboss
– no more windows server side
● Hypervisor based on RHEL 6
– Gives performance increase
● Local storage
● Reporing engine included
● WAN optimized SPICE

84. RHEV 3.0
● Power User portal
– Users can manage their own environment
● RESTful API
● Will be open sourced

85. RHEV adoption

86. OVA
● Open Virtualization Alliance
● Foster the adoption of KVM as an enterprise-
ready open virtualization solution
● Accelerate the emergence of an ecosystem of
third-party solutions around KVM.
● Encourage interoperability, promote best
practices, and highlight examples of customer
successes.

87. Ova : members

88. specvirt
● Virtualization benchmark (score@ 'n' VM's)
● KVM
– 3894@240
● VmWare ESX 4.1
– 3723@228

89. conclusions
● KVM is till young
● Adoption is growing
● Builds on Linux features (feature velocity)
– Selinux, Cgroups, Scheduler, ...
● A lot of community activity
● scalability
● Growing ECO system
● No vendor lockin
● OVA (open virtualization alliance)

90. Questions?

91. http://www.flickr.com/photos/kalavinka/4617897952/
http://www.flickr.com/photos/zakh/337938459/
http://docs.redhat.com/docs/en-
US/Red_Hat_Enterprise_Linux/6/html-
single/Virtualization/index.html
http://www.cyberciti.biz/faq/linux-kvm-vnc-for-guest-machine/
http://www.centos.org/docs/5/html/5.2/Virtualization/chap-
Virtualization-Managing_guests_with_virsh.html
http://heprc.phys.uvic.ca/sites/heprc.phys.uvic.ca/files/report
s/vliet-wtr.pdf

92. Grind out appliances

93. BoxGrinder : overview
● Create a VM “offline”
● Runs on JBOSS
● Use puppet/chef/.. to configure it further
● Move from Fedora to RHEL by adjusting the file
●