Nowadays there is significant diversity in Infrastructure as a Service (IaaS) clouds. The differences span from virtualization technology and hypervisors, through storage and network configuration, to the cloud management APIs. These differences make migration of a VM (or a set of VMs) from a private cloud into a public cloud, or between different public clouds, complicated or even impractical for many use-cases. HVX is a virtualization platform that enables complete abstraction of underlying cloud infrastructure from the application virtual machines. HVX allows deployment of existing VMs into the cloud without any modifications, mobility between the clouds and easy duplication of the entire deployment. HVX can be deployed on almost any existing IaaS cloud. Each instance of the HVX deployment packs in a nested hypervisor, virtual hardware, network and storage configuration. Combined with image store and management APIs, the HVX can be used for the creation of a virtual cloud that utilizes existing cloud provider infrastructure as the hardware rather than using physical servers, switches and storage.

1. HVX: Virtualizing The Cloud
Alex Fishman, Mike Rapoport, Evgeny Budilovsky, Izik Eidus
Ravello Systems

2. Virtual cloud on top of existing public clouds
➜ Nested hypervisor
➜ Overlay network on top of existing cloud provider’s
network
➜ Storage abstraction layer
➜ APIs for VM, network and storage provisioning

3. Virtual cloud
3
Public Cloud BPublic Cloud A
L2 guest VM
HW/net/stor
age
abstraction
L1 Cloud VM
public network
L2 overlay network
L2 guest VM
HW/net/stor
age
abstraction
L1 Cloud VM
L2 guest VM
HW/net/stor
age
abstraction
L1 Cloud VM

4. Motivation
➜ Migration of unmodified multi-VM applications
between different private and public clouds
› Prevent single provider lock-in
➜ Easy to build hybrid clouds
› Elasticity: using a public cloud provides infinite amount
of compute resources on-demand
› High availability
› Cost optimizations
4

5. Challenges of using public cloud
➜ Incompatible APIs
› EC2, OpenStack flavors, Azure
➜ Different virtualization technologies
› PV vs HVM
› Virtual HW exposed to VM
➜ Network topology
› Vary from provider to provider
› Often rigid and inflexible
➜ Storage
› Incompatible solutions
5

6. Solution: virtual private cloud on top of a public cloud
➜ Overlay private network on top of a cloud provider’s
network
➜ Virtualize cloud provider’s storage solutions
➜ Nested virtualization atop of cloud instances
6

7. Nested virtualization
➜ Nested virtualization is implemented by major
hypervisors
› Requires newer CPU models
› Intel VT-x/AMD-V extensions must be visible to L1 guest
VM
› Depends on particular host system configuration
› Experimental technology and not ready for the prime time
7

8. Public clouds do not support nested virtualization
➜ Huge existing install base
› Not easy to make changes
➜ Heterogeneous hardware and hypervisors
› Mix of older and newer hardware
› Different hypervisor versions
8
But…

9. HVX hypervisor
➜ HVX is a high performance
nested hypervisor
› Can run on top of any hypervisor
› Runs unmodified guest VMs
› Supports PV and HVM for bottom
virtualization layer
› Runs on top of Linux
HVX architecture
9
Unmodified guest
VM
Guest VM
hardware
HVX hypervisor
Linux
Cloud VM
hardware
Cloud VM

10. HVX highlights
➜ Implements its own set of virtual hardware
› Compatible with KVM, ESX, XEN
➜ Resource overcommit
› CPU and RAM overcommit can lower the costs of using
public cloud infrastructure for lite workloads
➜ Live migration
› HVX allows live VM migration between different public
cloud operators
10

11. Dynamic binary translation
➜ Translates guest binary code
into platform compatible code
› 32bit guest into 64 bit host
› Replaces privileged instructions
› CPU registers are shadowed
➜ HVX reads and compiles a
guest basic block and then
executes it
➜ At the end of each translated
code section there is a jump
back to HVX and the process
repeats
1111
Instruction
Instruction
Instruction
….
Jump to next
Tc Instruction
Tc Instruction
Tc Instruction
….
Jump to HVX
Original guest code Translated guest code
Next:
Instruction
Instruction
Instruction
….
Instruction
Next TC:
Tc Instruction
Tc Instruction
Tc Instruction
….
Jump to HVX
HVX compiles next
code section

12. ➜ Performance optimizations
› Translated code is cached and chained
› Fast shadow MMU implementation
› Direct execution or user space code (Ring 3)
› PV device implementation for fast IO
› Fast context switching between guest and host kernel
› Uses Linux for guest CPU scheduling and memory management
➜ Memory protection and data isolation
› Each guest runs in its own context and cannot access HVX or host
memory
› Guest resource usage (CPU/IO) can be throttled if needed
12
HVX implementation cont.

13. Network Layer
➜ Secure L2 overlay
network over cloud
operator’s L3 network
› UDP as a transport layer
› Can span across multiple
clouds
➜ Provides additional
services: DHCP, DNS and
Routing
› The services are fully
distributed: no single
point of failure
13

14. Image Store
➜ Supports multiple back-ends
› Amazon S3
› RackSpace CloudFiles
› NFS attached volumes
➜ Contains read-only snapshot images
› A VM image is a chain of snapshots
› The top snapshot contains differences from the base
snapshot
› Changes are written to the top snapshot stored locally on a
VM
› The top snapshot can be written back to the store
14

15. Guest VM storage
➜ HVX adds an abstraction layer above the physical
storage in the cloud
➜ Logical volumes are attached to guest VMs as local
block devices
15

16. Performance evaluation of HVX hypervisor
➜ Phoronix Test Suite
› apache, openssl, phpbench, pybench, pgbench
➜ timed kernel build
➜ iperf for network testing
Instance type CPUs Memory Virtualization
EC2: m1.large 2 7.5 GB Xen PV
EC2: m3.xlarge 4 15 GB Xen HVM
HP: standard.xlarge 4 16 GB KVM
HVX guest 2/4 4 GB HVX
16

17. Performance evaluation cont.
17

18. Conclusions and future work
➜ HVX is a versatile platform for the creation of a
virtual cloud spanning across public and private
clouds
➜ Future work
› Integration with OpenStack
› Run OpenStack compute nodes in a public cloud
› Connecting hSwitch to OpenStack Quantum component
› Create a hybrid OpenStack cloud by creating a bridge between
local data center and a public cloud
18