Hardware support for efficient virtualization

1. HARDWARE SUPPORT
FOR EFFICIENT
VIRTUALIZATION
Lennox Wu
1

2. Outline
• Classifications
• Processor virtualization
Two main Software-based solutions
• Challenges to virtualize Intel x86(software-only)
• Hardware-based Virtualization
•
Intel VT-x : x86
• Intel VT-I :Itanium (X)
• Intel EPT/AMD NPT
• AMD-V
• Sun SPARC (X)
• ARM Virtualization Extensions(X)
• IBM Power(X)
•
• I/O virtualization
• Intel VT-d
• AMD IOMMU(AMD-V)
• Intel VT-c
• Dose these techniques work?
2

3. Classifications
• VMM(virtual machine monitor) = hypervisor
• By techniques
Full Virtualization
• Paravirtualization
• Hardware Assisted Virtualization
•
• Robert P. Goldberg(Harvard University,1973)
•
Type 1/native/bare metal hypervisors
Hypervisors run directly on the host's hardware to control the hardware and to manage
guest operating systems.
• Oracle VM Server for SPARC, the Citrix XenServer, KVM, VMware ESX/ESXi, and Microsoft
Hyper-V hypervisor.
•
•
Type 2/hosted hypervisors
Hypervisors run within a conventional operating system environment.
• VMware Workstation/player, Microsoft Virtual PC and VirtualBox
•
3

4. TYPE1/TYPE2 hypervisors
4

5. PROCESSOR
VIRTUALIZATION
5

6. Two main Software-based
solutions(1)
• Full virtualization using binary translation
•
Transforming guest OS binaries on-the-fly
•
•
Guest applications don’t use privilege instructions
Pros
Support unmodified OSs (The only way of pure-software solutions)
• Offer best isolation and security
• Simplify migration and portability of guest OS
•
Cons: low performance
• Examples: VMware, MS Virtual PC, Virtual box
•
•
Disable HW virtualization
6

7. Full virtualization using binary
translation
7

8. •
Two main Software-based
OS assisted virtualization or paravirtualization
solutions(2) guest OSs help the VMM
OS assisted virtualization
•
•
•
paravirtualization refers to communication between the guest OS and the VMM to
improve performance and efficiency
Modify the guest OS to cooperate with the VMM
•
Modify the OS kernel to replace non-virtualizable instructions with hypercalls(the
functions provided by the VMM)
Pros: Offer higher performance
• Cons: Need the source code of an OS
• Example: Xen, KVM(*), VMware(*)
• (*) Vmware tool uses some paravirtualization techniques
•
optimize virtual device drivers
• time synchronization
• logging and guest shutdown.
• Vmxnet is a paravirtualized I/O device driver
•
8

9. OS assisted virtualization or
paravirtualization
9

10. KVM
KVM
10

11. Challenges to virtualize Intel
x86(software-only)(1/3)
• Ring Aliasing
•
Guest-OSes run at the Ring3
•
•
Original: OS:Ring 0, APP:Ring 3 (Ring0>ring3)
A guest OS can know its run level
• Address-Space Compression
VMM must use some of the guest’s virtual-address space to manage
transition between guest OS and VMM
• VMM’s address spaces must be protected
•
•
Guest could detect that it is running in a VM
11

12. Challenges to virtualize Intel
x86(software-only)(2/3)
• Non-Faulting Access to Privileged State
•
Some instructions should be intercepted by VMM do not involve faults
• Adverse Impact on Guest System Calls
•
VMM must emulate every system calls
• Interrupt Virtualization
A VMM may manage external interrupts and deny guest to control
interrupt masking
• Some OS frequently mask and unmask
•
•
VMM must process these requests.
12

13. Challenges to virtualize Intel
x86(software-only) (3/3)
• Ring Compression
•
Guest OS runs at the same privilege level as applications
•
The guest OS can’t protect guest applications
• Frequent Access to Privileged Resources
•
VMM should deny the accesses
• Address translation
•
Guest OS doesn’t know the physical address, so the VMM must
intercepted guest page table updates
13

14. Intel VT-x overview(1/4)
• VT=virtualization technology
•
Two new form of CPU operation
VMX root operation : for VMM
• VMX non-root operation: for guest-software
• Both forms of operation support all four privilege levels(Ring0~Ring3)
•
•
Guest OS can run at its intended privilege level
14

15. Two new form of CPU operation
15

16. Intel VT-x overview(2/4)
• Two new transitions
• VM entry
• VMX root operation (VMM) non-root operation(VM)
• VM exit
• VMX non-root operation (VM) root operation (VMM)
• Under VMX non-root operation, many
instructions/events cause VM exits
• configurable
16

17. Intel VT-x overview(3/4)
• VMCS (Virtual Machine Control Structure)
A new data structure includes guest-state area and host-state area
• VM entry: load the guest-state area and save the host-state area
• VM exit : load the host-state area and save the guest-state area
• The exiting conditions controlled by the VM-execution fields
• Switch the structure will switch the address space
•
17

18. Intel VT-x overview(4/4)
• VMCS supports interrupt virtualization
•
Determine the conditions of VM to cause VM exit
•
•
•
•
•
All interrupt
Whenever guest OS is ready to receive interrupts
Which exception?
Which port access attempts?
Which Model Specific Register access attempts?
18

19. Intel EPT / AMD NPT(1)
• EPT (Extended Page Tables)
•
•
“EPT provides performance gains of up to 48% for MMU-intensive
benchmarks and up to 600% for MMU-intensive microbenchmarks.” –
VMware
AMD’s nested page table (NPT) is similar to EPT
• A.k.a Rapid Virtualization Indexing (RVI)
•
“RVI provides performance gains of up to 42% for MMU-intensive
benchmarks and up to 500% for MMU-intensive microbenchmarks.” -VMware
19

20. Intel EPT / AMD NPT(2)
• Software MMU (software-only)
Hardware uses the shadow page table
• VMM must maintain the shadow page table
•
20

21. Intelmaintains guest page tables
EPT / AMD NPT(3)
Guest-OS
• Hardware MMU
•
VMM maintains PPN->MPN mappings in an additional level of page tables
• The hardware will find the LPN->MPN with the two pages
•
21

22. AMD-V(1/2)
• Tagged TLB
Add the ASID
• Hardware features that facilitate efficient switching between virtual
machines for better application responsiveness
•
•
Two new form of CPU operation
Host mode : for VMM (similar to Intel’s VMX root operation)
• Guest mode : for guest software (similar to Intel’s VMX non-root operation)
• new instructions
•
•
•
•
•
vmrun : host mode  guest mode
exit : guest mode  host mode
vmcall: it lets the operating system and VMM communicate directly
A new structure
•
Virtual Machine Control Block (VMCB)
•
Similar to Intel’s VMCS
22

23. AMD-V(2/2)
• Nested page table (NPT)/ Rapid Virtualization Indexing (RVI)
• VMM migration
•
Use the CPUID to identify the ability of the processor where the VMM
runs, and the VMM use the supported functions.
23

24. Hardware-base solution with VTx(1/2)
• Address-Space Compression
•
VM Exits / VM Entries change the linear address space
• Ring Aliasing
•
& Ring Compression
VT-x allows guest OS to run at its intended privilege level
• Nonfaulting Access to Privileged State
Either causes transition to VMM
• Or becomes unimportant to VMM
•
24

25. Hardware-base solution with VTx(2/2)
• Guest System Calls
•
a guest OS can run at privilege level 0
• Frequent Access to Privileged Resources
•
VT-x provides TPR shadow. VMM is only involved when the value drops
below the threshold VMM only processes the situation it cares.
25

26. Hardware Assisted Virtualization
of x86
26

27. I/O VIRTUALIZATION
27

28. Current I/O virtualization
techniques
• Emulation
The VMM supports virtual devices that guest OS can recognize
• The virtual device models are responsible to translate commands and data.
• Pros. No requirement to modify guest-OSs
• Cons. Low performance
•
• Paravirtualization
Modify the guest software (driver)to enhance the performance
• Pros. better performance
• Cons. Limited applicability. (modify need the source code)
•
• Direct assignment
Bind a specify device to a VM
• VMM allow the owning VM to connect directly
• Issue command (go) low overhead
• DMA? (back)
•
28

29. DMA on a virtualizing system
• DMA
Driver issue a packet consists of command, physical address, etc.
• DMA controller read/write data from/to the physical address
• Challenge?
•
A physical address that a Guest-OS knows is not really physical !
• The really physical address space is managed by the VMM
• The DMA controller will incorrectly write data to an address.
•
29

30. Intel VT-d(1/2)
• Need the support of the North bridge
• Two functions
•
Bind devices to a specify VM
•
•
DMA remapping
Interrupt virtualization
•
Interrupt remapping
• DMA remapping
DVA (DMA Virtual Address), GPA(Guest Physical Address), HPA(Host
Physical Address)
• A guest-OS issue a DMA request with DVA(=GPA)
• The VT-d hardware will translate the DVA to HPA
•
•
The concept: lookup tables
30

31. DMA remapping
31

32. Intel VT-d (2/2)
•
Interrupt Remapping
• Assign an interrupt attribute
•
•
Destination processor, vector, etc.
A VMM enables the interrupt requests from the I/O device to target the
physical CPUs running the appropriate virtual CPUs of the legacy VM
• AMD IOMMU is similar to Intel VT-d
32

33. Intel VT-c
• Virtualization Technology for Connectivity
•
Virtualization on devices
• A collection of technologies that improve the performance of
network I/O on a virtualized system
• VT-c is comprised of two components
•
VMDq (Virtual Machine Device Queues)
A hardware-base enhancement
• Target: throughput
•
•
VMDc (Virtual Machine Direct Connect)
Virtualizing physical I/O ports of a network controller into multiple virtual I/O
ports, and then to map the virtual ports to individual VMs
• Target :VT-x + VT-d + VT-c  nearly native performance
•
33

34. Why VMDq?
34

35. 35

36. VMDc
36

37. DO THESE
TECHNIQUES WORK?
37

38. Ubuntu 11.10: Xen vs. KVM vs.
VirtualBox(1)
38

39. Ubuntu 11.10: Xen vs. KVM vs.
VirtualBox(2)
39

40. Ubuntu 11.10: Xen vs. KVM vs.
VirtualBox(3)
40

41. Ubuntu 12.10: KVM vs. Xen (1)
41

42. Ubuntu 12.10: KVM vs. Xen (2)
42

43. Ubuntu 12.10: KVM vs. Xen (3)
43

44. Ubuntu 12.10: KVM vs. Xen (4)
44

45. Ubuntu 12.04 KVM/Xen
Virtualization: Intel vs. AMD(1)
• Ubuntu 12.04 LTS, an Intel Core i7 3960X "Sandy Bridge" Extreme
Edition and AMD FX-8150 "Bulldozer" systems were used.
45

46. Ubuntu 12.04 KVM/Xen
Virtualization: Intel vs. AMD(2)
46

47. Intel Ivy Bridge Linux
Virtualization Performance(1)
47

48. Intel Ivy Bridge Linux
Virtualization Performance(2)
48

49. Summarization of Hardware
Assisted Virtualization
• Hardware provides some mechanisms to reduce overheads of
virtualization to improve performance
• Pros.
The highest performance in theory (a counter example, 2006 VMware)
• Support unmodified Oss
• Simplify the development of VMM
•
• Cons.
•
Need newer processors
• Example
•
KVM(basic requirements)
49

50. References
•
Performance Evaluation of Intel EPT Hardware Assist, VMware
•
I/O Virtualization and AMD's IOMMU
•
•
Processor-Based Virtualization, AMD64 Style, Part I
•
•
http://developer.amd.com/documentation/articles/pages/630200614.aspx
Processor-Based Virtualization, AMD64 Style, Part II
•
•
http://developer.amd.com/documentation/articles/pages/892006101.aspx
http://developer.amd.com/documentation/articles/pages/630200615.aspx
Intel technology Journal, vol 10, issue 3, 2006
Intel virtualization technology: Hardware Support for Efficient processor virtualization
• Intel virtualization technology for Directed I/O
•
•
ARM virtualization Extension Architecture Specification
•
A Comparison of software and hardware techniques for x86 virtualization,Vmware
•
http://www.intel.com/network/connectivity/solutions/vmdc.htm
•
http://www.intel.com/network/connectivity/solutions/vmdq.htm
•
http://software.intel.com/en-us/blogs/2009/09/30/understanding-vt-c-virtualizationtechnology-for-connectivity/
50

51. References
• Ubuntu 11.10: Xen vs. KVM vs. VirtualBox
http://www.phoronix.com/scan.php?page=article&item=ubuntu_11
10_xenkvm&num=1
• Ubuntu 12.04 KVM/Xen Virtualization: Intel vs. AMD
http://www.phoronix.com/scan.php?page=article&item=ubuntu_12
04_virt&num=1
• Intel Ivy Bridge Linux Virtualization Performance
http://www.phoronix.com/scan.php?page=article&item=intel_iv
y_virtualization&num=5
• http://en.wikipedia.org/wiki/Hypervisor
51

52. Q&A
52

53. THANK YOU
53