Nested Virtualization Update from Intel

1. Nested Virtualization Update From IntelXiantao Zhang, Eddie DongIntel Corporation

2. Legal DisclaimerINFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NOLICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUALPROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMSAND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER,AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OFINTEL® PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR APARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OROTHER INTELLECTUAL PROPERTY RIGHT. INTEL PRODUCTS ARE NOT INTENDED FOR USE INMEDICAL, LIFE SAVING, OR LIFE SUSTAINING APPLICATIONS.Intel may make changes to specifications and product descriptions at any time, without notice.All products, dates, and figures specified are preliminary based on current expectations, and are subject tochange without notice.Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, whichmay cause the product to deviate from published specifications. Current characterized errata are available onrequest.Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in theUnited States and other countries.*Other names and brands may be claimed as the property of others.Copyright © 2012 Intel Corporation.

3. Agenda• Motivation and Goals• History −Nested VMX Architecture −Previous status• Latest status and new features −Stability Enhancement −Virtual EPT −Virtual VT-d• Preliminary Performance• Call to Action 3

4. Motivation and Goals• Why nested virtualization? − Ordinary OS are adopting VMX now −Windows 7 XP compatibility mode −Windows 8 Hyper-V − Other Commercial VMMs requires VMX for Guest better performance − vmware vmm − Anti-virus software depends on VMX Guest − McAfee Deep Defender• What is the goal ? VMM − To make VMX-based system software run smoothly in a Xen guest. VMM Hardware Platform with VMX Enabled 4

5. Agenda• Motivation and Goals• History −Nested VMX Architecture −Previous status• Latest status and new features −Stability Enhancements −Virtual EPT −Virtual VT-d• Preliminary Performance• Call to Action 5

6. Nested VMX Architecture VM entry/exit Virtual VM entry/exit Nested Guest L2Dom0 HVM Guest Nested L1 vVMCS VMM Shadowing Shadow VirtualXen VMM VMCS VMCS EPT L0 Virtual Virtual VMX VT-d VMX-Enabled Platform 6(VT-d, EPT etc.)

7. History• Nested VMX update @ Xen Summit Asia (Nov. 2009) − Nested VMX design is presented − Showed Initial Status −Nested guest can boot up to BIOS early stage with limitations − single vCPU/single nested guest/ No vCPU migration• Refined nested VMX support was pushed into upstream − Support multiple nested guests − Also includes supporing SMP nested guests• However, experimental & preliminary support − Very limited configurations can work −“KVM on Xen”, Linux guest can successfully boot up −“Xen on Xen” does not work − No virtual VT-d, virtual EPT 7

8. Previous Status − Only one combination can work L2 Guest OSL0-VMM L1-VMM 32Bit PAE OS 64Bit OS Win2012 Ubuntu RHEL6.0 RHEL5.4 Win7 RHEL6.0 Win7 Server 12.04 Xen X X X X X X X Xen KVM X √ X X X X X 8

10. Stability Enhancement − Greatly enhanced stability, with several critical bugs fixed! L2 Guest OS L2 Guest OS(SMP)L0-VMM L1-VMM 32Bit PAE OS 64Bit OS Win2012 Ubuntu RHEL6.0 RHEL5.4 Win7 RHEL6.3 RHEL6.0 Win7 Server 12.04 Xen X √ X √ X √ X √ X √ X √ X √ Xen KVM X √ √ X √ X √ X √ X √ X √ 10

11. Performance Without Optimizations 1 L2 Guest2(Xen on Xen)0.90.8 L1 Guest0.70.6 Native0.50.4 Platform: SNB-EP0.3 OS: RHEL6.3 Guest MEM:4GB Memory0.2 CPU: 2 VCPU0.1 0 SpecJBB Unixbench Kernel Build CPU-INT 11

13. VM entry/exit Virtual EPT Architecture Virtual VM entry/exitL2 L2 GuestL1 vVMCS Shadowing L2-GPA-> L1 GPA L1 VMM L1 EPT ShadowingL0 VMCS-L1 sVMCS-L2 L0 EPT L1-GPA-> HPA Shadow EPT L0 VMM L2-GPA->HPA Switch to Shadow EPT @ virtual vmentry 13

14. Virtual EPT: Using EPT Shadowing• No write-protection to L1-EPT (Guest EPT paging structure) − Flexibility is good.• Trap-and-emulate guest’s INVEPT − Update the shadow EPT entries• Better SMP Scalability − No global lock is required• Requires page-level INVEPT − Individual address invalidation 14

15. Enhanced INVEPT Instruction for Virtual EPT• INVEPT limitations − No Individual address invalidation −Only single context and all context invalidation • Little performance impact, however, hurt nested performance sharply! −Has to drop shadow EPT table for L1’s each INVEPT(with single context) • Performance loss if frequent INVEPT in VMM • For example, KVM• Enhance it in Software Way − Add Individual address invalidation for virtual EPT −Expose it to nested VMM through PV approach − Need to enhance VMMs −Easy implementation for Xen and VMM• Benefits − Reduce frequent shadow EPT paging structure flush 15

16. Performance Evaluation For Virtual EPT7654 w/o virtual EPT3 with virtual EPT210 Kernel Build SpecJBB Netperf CPU-INT 16

18. Virtual VT-d: Expose VT-d Capability to L1VMM• I/O performance for L2 guest is very slow − Due to extremely long device emulation path through all the way to L1 & L0 VMMs• How to fix that? − Present virtual VT-d engine to L1 VMM − So, device can be directly assigned to L2 guest −High I/O performance, because of minimum VMM intervention.• Must-to-have features in Virtual VT-d − DMA Remapping & Queue Invalidation: Exposed − Interrupt remapping: Not Exposed 18

19. Virtual VT-d Architecture Nested Guest L2 Domain 0 Guest view of VT-d DMA Engine L1 Bus 0 DevQ,FunyDev Q: Qemu device Qemu Dev P: PT device HVM DevP,Funn loader Nested VMM VT-d page table Hypercall Hypercall Shadowing Virtual VT-d Hardware VT-d vDMAR vQI IO TLB Bus 0 Xen L0 Dev M FunN Bus N Device ATC VMM Shadow VT-d page table Hypervisor 19

20. Two types of guest devices• Pass through device −DMA (IOVA->GPA) is handled by hardware VT-d engine −Remap guest root/context structure −Use physical remapping table to emulate guest remapping table − IOVA -> L0 HPA, + audit (use a dummy page for Out of Bound gpn) − Maybe cached by IOTLB and ATC −IOTLB/Context Cache Synchronization −Track guest invalidation of IOTLB − Invalidate physical IOTLB, and may invalidate ATC as well if the device has ATC −Track guest invalidation of Context Cache• Qemu device −DMA (IOVA->PA remapping) is emulated by Qemu −2 Options: Caching the remapping table, or No-Caching −Starting from simple solution: No caching −Qemu device is already slow 20

21. Performance Evaluation of virtual VT-d 1100 Bandwidth of Nested Guest Ideal Bandwidth 1000 900 800Mb/s 700 600 500 400 300 200 100 0 TCP Send TCP Receive UDP Send UDP Receive Iperf testing with the assigned NIC to nested Guest Bandwidth is good enough! 21

22. Latency Evaluation of virtual VT-dms Latency120 Latency100 80 60 40 20 0 Native Nested Guest Still have room to tune Latency 22

24. Preliminary Performance Based on Xen #254671.2 10.8 L1 Guest0.60.4 L2 Guest0.2 0 Platform: SNB-EP OS: RHEL5.4 Guest MEM:2GB Memory CPU: 2 VCPU 24

26. Call to Action• Support more L1 VMMs − McAfee Deep Defender − VMware VMM − Hyper-V − Virtual Box• Virtual APIC-V − New Features for Interrupt/APIC Virtualization are coming − For more information, please come to Nakajima Jun’s talk “Intel Update” this afternoon. − Improve interrupt virtualization efficiency for both L1 and L2• Performance Tuning 26

27. Reference• Nested Virtualization on Xen − Qing He: − Xen Summit 2009: http://xen.org/xensummit/xensummit_fall_2009.html• Virtual APIC-V − Jun Nakajima: Intel Update − Xen Summit 2012: http://www.xen.org/xensummit/xs12na_talks/T10.html 27

28. Questions?• Or contact xiantao.zhang@intel.com 28

Nested Virtualization Update from Intel

The Linux Foundation

Nested Virtualization Update from Intel