Xen Summit Asia 2009

1. ®
ChinaV: Experiences on Virtualization Technology
Hai Jin,
Huazhong University of Science and Technology

2. Outline ®
• Introduction of ChinaV Project
• Experiences on Virtualization Technology
• Conclusions

3. What is ChinaV ®
http://grid.hust.edu.cn/973
• Research on Fundamental Theory and
Approach of Computing System
Virtualization, supported by National 973
Basic Research Program of China under
grant No.2007CB310900
• Started from 2007 to 2011 with total budget
RMB 26M

4. Visions ®
http://grid.hust.edu.cn/973
• Virtualized Resources Environment
– Combine or divide resources: good granularity and
transparence
• Virtualized Tasks Environment
– Build task executing environment on-demand: high
utilization and efficiency
• Virtualized User Environment
– Desktop virtualization: high convenience, good user
experiences

5. Missions ®
http://grid.hust.edu.cn/973
• Theoretical model and architecture of the virtualized computing
system
• Single-dimensional system resource virtualization
• Multi-dimensional system resource virtualization
• Pervasive computing environment of virtualized system
• Security and trusted scheme of the virtualized computing
system
• Theory and approach of evaluating virtualized computing
system
• High performance based virtualization technology
• Application of virtualized simulation system

6. ®
Research Team http://grid.hust.edu.cn/973

7. Outline ®
• Introduction of ChinaV Project
• Experiences on Virtualization
Technology
– Live Migration
– Power Management
– Memory Virtualization
– Desktop Virtualization
• Conclusions

8. Outline ®
• Introduction of ChinaV Project
• Experiences on Virtualization Technology
– Live Migration
– Power Management
– Memory Virtualization
– Desktop Virtualization
• Conclusions

9. ®
CR/TR-Motion: A Novel VM Migration Approach
• Revirt is adopted
• Checkpointing/recovery with trace/replay
technology are used to provide fast and
transparent live VM migration
• We orchestrate the running source and target
VM with execution trace logged on the source
host
H. Liu, H. Jin, X. Liao, L. Hu, and C. Yu, “Live Migration of Virtual Machine Based on Full System Trace and Replay”,
Proceedings of the 18th International Symposium on High Performance Distributed Computing (HPDC'09), ACM Press, June 11-
13, 2009, Munich, Germany, pp.101-110

10. CR/TR-Motion System Structure ®

11. CR/TR-Motion: Migration Process ®
Checkpoint
A log1 Checkpoint
B
Round 1 log2 VM Recovery
Round 2 Replay log1
…… log3
Round n …… Waiting and chasing phase ……
Transfer log n
Stop and copy
Replay log n
Take over A

12. CR/TR-Motion: Migration Downtime ®
• Our approach reduced migration downtime by 72.4% in average compared to
pre-copy approach
• Our approach reduces the total migration
300
CR/TR-Motion time by 31.5% in average compared to
Pre-copy
250
Pre-copy
200 100
Downtime(ms)
CR/TR-Motion
90
150
Pre-copy
80
To l m ra n tim (s)
100
70
e
50 60
ta ig tio
0 50
Daily use Kernel-build Static web Dynamic UnixBench 40
app web app
30
20
10
0
Daily us e Kernel-build Static web Dynamic UnixBench
app web app

13. ®
CR/TR-Motion: Total Data Transferred
• CR/TR-Motion reduces 900
CR/TR-Motion
T ta D ta T n rre (M )
800
B
Pre-copy
700
synchronization traffic
o l a ra sfe d
600
500
by 95.9% in average 400
300
200
• This improvement 100
0
brings great benefit Daily use Kernel-build Static web
app
Dynamic
web app
UnixBench
when our migration
scheme is applied in daily use
kernel-build
0.48 (0.04)
0.53 (0.06)
38.54 (2.1)
152.44 (8.2)
98.8%
99.6%
low-bandwidth WANs static web app
dynamic web app
8.34 (0.21)
36.4 (0.96)
228.99 (9.4)
288.05 (12.2)
96.4%
87.4%
unixbench 2.59 (0.22) 113.38 (6.4) 97.7%

14. Outline ®
• Introduction of ChinaV Project
• Experiences on Virtualization Technology
– Live Migration
– Power Management
– Memory Virtualization
– Desktop Virtualization
• Conclusions

15. Power Management - Motivation ®
• Reduce power consumption with little performance
penalty
• User requirement is various
– Server – Average power consumption should fit the budget
– Desktop – User experience should be kept
– Laptop – Prolong battery lifetime
• Virtualization brings challenges
– Guest OS is blind to the hardware features
– VMM lacks the device PM ability – it has no device drivers

16. Design of ClientVisor ®
• Focus on desktop virtualization
– VMs are asymmetric
– Hardware power features can be exposed to VM
• Dom0 – Domain 0, the control domain
• SOS – Service OS, background domain for
special tasks. e.g., network packet filtering
• COS – Capability OS, primary domain
interacted with users
H. Chen, H. Jin, Z. Shao, K. Yu, and K. Tian, “ClientVisor: Leverage COTS OS Functionalities for Power Management in Virtualized
Desktop Environment”, Proceedings of the 2009 ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments
(VEE’09), ACM Press, March 11-13, 2009, Washington, DC, USA, pp.130-141.

17. ®
Design of ClientVisor
Frontend A: Px operation
Driver COS (primary user domain)
Backend C B: Cx operation
SOS Driver Domain0 Device
Device (control domain) Driver OSPM
VA Driver C: Dx operation
C C B
Xen VMM Coordination Logic A D: Cx operation after
coordination
D
E E: Dx operation after
Physical Platform Devices CPU coordination

18. Design of ClientVisor ®
• Basic instruments – What guest OS does?
– Processor PM instruments – working state PM
(P-state scaling) & idle state PM (C-state
transition)
– Device PM instruments – D-state transition
• Interception policies – What VMM does?
– Passing-through – for P-state operation
– Coordination – for C-state & D-state operation

19. ®
ClientVisor
• Preliminary of passing-through – Root
Exposing hardware power features
– ACPI tables Bridge Endpoint Bridge
– CPUID
– Device hierarchy
Endpoint Endpoint Endpoint Endpoint

20. Performance Evaluation ®
• Static power consumption • Dynamic power
– Leave the whole system in idle
18
consumption 30.00
25.00 23.50
28.43
26.85 26.27 26.21
16.71
15.36
16
– SPECpower_ssj2008 is
20.00
Power (W)
13.34 13.01
14
15.00
12 10.91
used as workload
Power (W)
10 10.00
8 5.00
6
4
0.00
Native Xen CV/Orig CV/Cx_op t CV/Cx_Timer_op t
2
0 (a) Overall
Native Xen CV/Orig CV/Cx_op t CV/Cx_Timer_op t
40,000
35,000
40
Cx mapping optimization – Change Cx operation
35
ops)
30,000 of port I/O way to MWAIT way
erform nce(ssj_
30
o er )
P w (W
25,000
20,000 25
Timer optimization – Disable some timer
a
15,000
20
10,000
handlers when CPU resides in Cx
P
15
5,000
0 10
100% 90% 80% 70% 60% 50%
Load Level
40% 30% 20% 10% 0%
Balance of power and
Native Xen CV/Orig CV/Cx_opt
CV/Cx_Timer_opt
CV/Cx_opt
Native
CV/Cx_Timer_opt
Xen CV/Orig
performance

21. Outline ®
• Introduction of ChinaV Project
• Experiences on Virtualization Technology
– Live Migration
– Power Management
– Memory Virtualization
– Desktop Virtualization
• Conclusions

22. ®
Dynamic Memory Balancing for Virtual Machines
• Motivation
– Allocating appropriate machine memory to a VM is hard
• Memory requirement varies during running
• OS only reports the amount of used/free memory
• The amount of actively used memory is more important
– If we know the relationship between memory allocation size and
performance gain/loss
• Idle or inactive memory can be reclaimed without notable performance
loss
• Better memory resource utilization
– Ballooning
• The amount to increase/decrease is typically specified manually

23. ®
Dynamic Memory Balancing for Virtual Machines
• Dynamic memory balancing
– LRU-based predictor
– Memory growth prediction
– Automatic memory resizing inflate/deflate
VM1 VM2 Controller
balancer
WSS
Mon Mon estimator
Data store
LRU Hist. LRU Hist. VMM
W. Zhao and Z. Wang, “Dynamic Memory Balancing for Virtual Machines”, Proceedings of the 2009 ACM SIGPLAN/SIGOPS
International Conference on Virtual Execution Environments (VEE’09), ACM Press, March 11-13, 2009, Washington, DC, USA, pp.21-
30.

24. ®
Dynamic Memory Balancing for Virtual Machines
• Estimation Accuracy (within Xen)
– VM is allocated with 214MB
Monotonic (40 ~ 170 MB) Random (40 ~ 170 MB)

25. Outline ®
• Introduction of ChinaV Project
• Experiences on Virtualization Technology
– Live Migration
– Power Management
– Memory Virtualization
– Desktop Virtualization
• Conclusions

26. Challenge of Desktop Virtualization ®
• User experience
– Fast, convenient, mobility
• Security
– Safeguard user private data
• Stability
– Reliability of the virtual desktop environment
• Serviceability
– Efficient use of CPU and memory resources
X. Liao, H. Jin, L. Hu, and H. Liu, “Towards Virtualized Desktop Environment”, Concurrency and
Computation: Practice and Experience, John Wiley & Sons, Ltd (accepted)

27. ®
System Architecture of Virtual Desktop
Data Server
APP Server
……
Xen Xen
server server
Internet
Virtualized
VCM PC
Domain 0 Domain U
Thin PDA
Xen
Client

28. ®
Save & Restore (Checkpointing)
• Multi-VM collaborative save & restore
– Recoverable long-running desktop applications
– User environment mobility
– High availability
• Multi-host checkpointing
– Checkpoint synchronization (Lamport clocks)
– Transparent rolling checkpoints (Copy-on-write)
– Memory image saving optimization

29. Virtual Appliance ®
App Server
• USB devices and
printers on the
client can be remote desktop
Network delivering
accessed by the
remote access mount
application on a USB device
local network or
client Plug in
the Internet

30. ®
VM Life Cycle Management
• Role-based life cycle monitor
scheme
• VM suspending management
• VM process priority policy
• VM template life cycle
management
• VM checkpoint life cycle
management

31. ®
All-in-one Desktop Environment

32. Outline ®
• Introduction of ChinaV Project
• Experiences on Virtualization Technology
• Conclusions

33. Conclusions ®
• As the technology base of cloud computing, virtualization technology
provide
– Support new architectures, devices
– High Utilization of IT facilities
– High Manageability
– Highly secure and isolate guaranteed environment
– Maintain good user experiences
• Challenges still exist in virtualization technology
– Scheduling
– Live Migration
– Power Management
– Memory/IO Virtualization
– Desktop Virtualization
– ……

34. ®
Thank you!