The document discusses increasing VM density on XenServer. It outlines various hard limits on VM density imposed by factors like the number of dom0 event channels. It analyzes how these limits were addressed in XenServer 6.1 and 6.2 through techniques like increasing the number of event channels and minor numbers. It also examines soft limits caused by high CPU usage from components like xenstored and qemu that can impact density. The goal is to understand current limits and ways to remove barriers to scaling density with hardware.

1. Increasing XenServer’s VM density
Jonathan Davies,
XenServer System Performance Lead
XenServer Engineering, Citrix
Cambridge, UK
24 Oct 2013
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
1 / 34

2. Outline
1
Scalability expectations
2
Hard limits
3
Soft limits
4
Benchmarks
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
2 / 34

3. Scalability expectations
Outline
1
Scalability expectations
2
Hard limits
3
Soft limits
4
Benchmarks
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
3 / 34

4. Scalability expectations
Users expect VM density to scale with hardware
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
4 / 34

5. Scalability expectations
Users expect VM density to scale with hardware
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
5 / 34

6. Scalability expectations
Users get upset when. . .
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
6 / 34

7. Scalability expectations
XenServer’s VM density scalability
hard density
limit :-(
hardware's
theoretical capacity
XS 6.1
(and earlier)
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
7 / 34

8. Scalability expectations
XenServer’s VM density scalability
hard density
limit :-(
hardware's
theoretical capacity
XS 6.1
(and earlier)
XS 6.2
practical density limit
(depending on nature of VMs)
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
hard density
limit :-)
24 Oct 2013
8 / 34

9. Hard limits
Outline
1
Scalability expectations
2
Hard limits
3
Soft limits
4
Benchmarks
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
9 / 34

10. Hard limits
Enumerated causes of limitations
Hard limit 1: dom0 event channels
Cause of limitation XenServer uses a 32-bit dom0
This means 1,024 dom0 event channels
#define MAX_EVTCHNS(d)
(BITS_PER_EVTCHN_WORD(d) *
BITS_PER_EVTCHN_WORD(d))
Various VM functions use a dom0 event channel
VM density hard limit 225 VMs per host (PV with 1 vCPU, 1 VIF, 1 VBD)
150 VMs per host (HVM with 1 vCPU, 1 VIF, 3 VBDs)
Mitigation for XS 6.2 Hack for dom0 to enjoy 4,096 event channels
→ 800 VMs per host (PV with 1 vCPU, 1 VIF, 1 VBD)
→ 570 VMs per host (HVM with 1 vCPU, 1 VIF, 3 VBDs)
Mitigation for future Change the ABI to provide unlimited event channels
→ this would remove the limit
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
10 / 34

11. Hard limits
Enumerated causes of limitations
Hard limit 2: blktap2 device minor numbers
Cause of limitation blktap2 only supports up to 1,024 minor numbers
(despite the kernel allowing up to 1,048,576)
#define MAX_BLKTAP_DEVICE
1024
Each virtual block device requires one device
VM density hard limit 341 VMs per host (with 3 disks per VM)
Mitigation for XS 6.2 Double this constant to 2,048
→ 682 VMs per host (with 3 disks per VM)
Mitigation for future Move away from blktap2 altogether?
→ this would remove the limit
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
11 / 34

12. Hard limits
Enumerated causes of limitations
Hard limit 3: number of aio requests
Cause of limitation Each blktap2 instance creates an asynchronous I/O
context for receiving 402 events.
Default system-wide number of aio requests was
444,416 in XS 6.1.
VM density hard limit 368 VMs per host (with 3 disks per VM)
Mitigation for XS 6.2 Set fs.aio-max-nr to 1,048,576
→ 869 VMs per host (with 3 disks per VM)
Mitigation for future Increase fs.aio-max-nr further
or use storage driver domains
→ this would remove the limit
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
12 / 34

13. Hard limits
Enumerated causes of limitations
Hard limit 4: dom0 grant references
Cause of limitation Windows VMs use receive-side copy (RSC) by default in XS 6.1.
netback allocates (at least) 22 grant-table entries per
virtual interface for RSC.
dom0 had a total of 8,192 grant-table entries in
XS 6.1.
VM density hard limit 372 VMs per host (with 1 interface per VM)
Mitigation for XS 6.2 Don’t use RSC in Windows VMs anyway
→ this removes the limit
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
13 / 34

14. Hard limits
Enumerated causes of limitations
Hard limit 5: connections to xenstored
Cause of limitation xenstored uses select(2), which can only listen on
1,024 file descriptors.
#define __FD_SETSIZE
1024
qemu opens 3 file descriptors to xenstored.
VM density hard limit 333 VMs per host (HVM)
Mitigation for XS 6.2 Make two qemu watches share a connection
→ 500 VMs per host (HVM)
Mitigation for future Upstream qemu doesn’t connect to xenstored
→ this will remove the limit
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
14 / 34

15. Hard limits
Enumerated causes of limitations
Hard limit 6: connections to consoled
Cause of limitation Similarly, consoled uses select(2)
Each PV domain opens 3 fds to consoled
VM density hard limit 341 VMs per host (PV)
Mitigation for XS 6.2 Use poll(2) rather than select(2) in consoled
→ this removes the limit
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
15 / 34

16. Hard limits
Enumerated causes of limitations
Hard limit 7: dom0 low memory
Cause of limitation Each running VM eats about 1MB of dom0 lowmem
VM density hard limit around 650 VMs per host
Mitigation for future Use a 64-bit dom0
→ this will remove the limit
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
16 / 34

17. Hard limits
Summary of hard limits
Summary of hard limits
Limits on number of HVM guests with 1 vCPU, 1 VBD, 1 VIF (with PV drivers)
Limitation
dom0 event channels
blktap minor numbers
aio requests
dom0 grant references
xenstored connections
consoled connections
dom0 low memory
Overall limit
Limited by
Jonathan Davies (Citrix)
XS 6.1
225
1024
1105
372
333
no limit
650
225
event channels
XS 6.2
800
2048
2608
no limit
500
no limit
650
500
xenstored
Increasing XenServer’s VM density
Future
no limit
no limit
no limit
no limit
no limit
no limit
no limit
very high
something else!
24 Oct 2013
17 / 34

18. Hard limits
Summary of hard limits
Summary of hard limits
Limits on number of HVM guests with 1 vCPU, 3 VBDs, 1 VIF (with PV drivers)
Limitation
dom0 event channels
blktap minor numbers
aio requests
dom0 grant references
xenstored connections
consoled connections
dom0 low memory
Overall limit
Limited by
Jonathan Davies (Citrix)
XS 6.1
150
341
368
372
333
no limit
650
150
event channels
XS 6.2
570
682
869
no limit
500
no limit
650
500
xenstored
Increasing XenServer’s VM density
Future
no limit
no limit
no limit
no limit
no limit
no limit
no limit
very high
something else!
24 Oct 2013
18 / 34

19. Hard limits
Summary of hard limits
Summary of hard limits
Limits on number of PV guests with 1 vCPU, 1 VBD, 1 VIF
Limitation
dom0 event channels
blktap minor numbers
aio requests
dom0 grant references
xenstored connections
consoled connections
dom0 low memory
Overall limit
Limited by
Jonathan Davies (Citrix)
XS 6.1
225
1024
368
no limit
no limit
341
650
225
event channels
XS 6.2
1000
2048
869
no limit
no limit
no limit
650
650
dom0 lowmem
Increasing XenServer’s VM density
Future
no limit
no limit
no limit
no limit
no limit
no limit
no limit
very high
something else!
24 Oct 2013
19 / 34

20. Hard limits
Summary of hard limits
500 Windows VMs on a host
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
20 / 34

21. Soft limits
Outline
1
Scalability expectations
2
Hard limits
3
Soft limits
4
Benchmarks
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
21 / 34

22. Soft limits
xenstored
High dom0 CPU utilisation by xenstored
top - 16:29:33 up 36 min, 1 user, load average: 0.80, 0.56, 0.47
Tasks: 132 total,
1 running, 131 sleeping,
0 stopped,
0 zombie
Cpu(s): 40.1%us, 40.0%sy, 0.0%ni, 17.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem:
4186504k total,
443480k used, 3743024k free,
23696k buffers
Swap:
524280k total,
0k used,
524280k free,
132504k cached
PID
7339
6665
7225
7269
7413
7283
10938
16403
1
2
3
USER
root
root
root
root
root
root
root
root
root
root
root
PR
20
20
20
20
20
20
20
20
20
20
RT
Jonathan Davies (Citrix)
NI VIRT RES SHR S %CPU %MEM
0 6732 2240 840 S 80.2 0.1
0 4344 2636 584 S 0.4 0.1
0 48892 5356 1736 S 0.3 0.1
0 23704 3684 1308 S 0.3 0.1
0 195m 21m 8932 S 0.3 0.5
0 7492 4860 1200 S 0.3 0.1
0 29808 1856 956 S 0.3 0.0
0 2428 1104 824 R 0.3 0.0
0 2164 656 564 S 0.0 0.0
0
0
0
0 S 0.0 0.0
0
0
0
0 S 0.0 0.0
Increasing XenServer’s VM density
TIME+
0:10.22
0:04.03
0:03.35
0:03.47
0:10.28
0:08.65
0:00.40
0:02.31
0:00.83
0:00.00
0:00.01
COMMAND
xenstored
fe
xcp-rrdd
xcp-rrdd-iostat
xapi
xcp-rrdd-xenpm
v6d
top
init
kthreadd
migration/0
24 Oct 2013
22 / 34

23. Soft limits
xenstored
High dom0 CPU utilisation by xenstored
dom0 vCPUs
domU vCPUs
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
23 / 34

24. Soft limits
xenstored
High dom0 CPU utilisation by xenstored
xenstored's dom0 vCPU
other dom0 vCPUs
domU vCPUs
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
24 / 34

25. Soft limits
xenstored
High dom0 CPU utilisation by xenstored
Cause of limitation xenstored CPU utilisation bottleneck
Mitigation for XS 6.2 Reduce xenstore use by XenServer’s toolstack:
remove some spurious writes
replace polling with watching
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
25 / 34

26. Soft limits
qemu
High dom0 CPU utilisation due to qemu
top - 16:40:27 up 2:07, 1 user, load average: 89.62, 87.22, 76.90
Tasks: 1015 total, 65 running, 950 sleeping,
0 stopped,
0 zombie
Cpu(s): 23.4%us, 55.5%sy, 0.0%ni, 4.8%id, 0.0%wa, 0.0%hi, 15.4%si, 0.5%st
Mem:
4180480k total, 1615840k used, 2564640k free,
3804k buffers
Swap:
524280k total,
0k used,
524280k free,
122852k cached
PID
7143
6653
16496
16970
997
3195
3497
3562
3993
7597
8150
8679
8974
11937
12545
14053
17752
USER
root
root
tcpdump
root
65583
65684
65656
65685
65546
65659
65550
65627
65661
root
65556
65598
65567
PR NI VIRT RES SHR S
20
0
0
0
0 R
10 -10 12264 7796 1152 R
20
0 5508 2132 1248 R
20
0 2952 1552 736 R
20
0 24696 4732 1572 S
20
0 24632 4736 1572 S
20
0 24760 4740 1576 R
20
0 24696 4732 1572 S
20
0 24888 4744 1580 S
20
0 24632 4736 1576 S
20
0 24760 4740 1580 R
20
0 24632 4740 1576 R
20
0 24568 4736 1572 S
20
0
0
0
0 S
20
0 24824 4748 1584 S
20
0 24760 4736 1576 S
20
0 24952 4740 1576 S
Jonathan Davies (Citrix)
%CPU %MEM
33.9 0.0
31.8 0.2
10.5 0.1
6.3 0.0
3.1 0.1
3.1 0.1
3.1 0.1
3.1 0.1
3.1 0.1
3.1 0.1
3.1 0.1
3.1 0.1
3.1 0.1
3.1 0.0
3.1 0.1
3.1 0.1
3.1 0.1
Increasing XenServer’s VM density
TIME+
17:21.63
36:14.34
5:35.12
0:00.11
0:56.30
0:27.34
0:28.65
0:26.97
0:53.19
0:28.86
0:51.71
0:31.18
0:27.97
1:12.92
0:58.46
0:31.33
0:56.82
COMMAND
rpciod/0
ovs-vswitchd
tcpdump
top
qemu-dm
qemu-dm
qemu-dm
qemu-dm
qemu-dm
qemu-dm
qemu-dm
qemu-dm
qemu-dm
nfsiod
qemu-dm
qemu-dm
qemu-dm
24 Oct 2013
26 / 34

27. Soft limits
qemu
qemu burning dom0 CPU
200 idle Windows guests,
each qemu utilising 3% of a CPU
means
6 dom0 vCPUs wasted!
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
27 / 34

28. Soft limits
qemu
What is qemu busy doing?
Emulated device
USB
CD-ROM
Buffered I/O & RTC timer
Parallel port
Serial port
VNC
qemu monitor
qemu events per VM per second
221
38
13
1
1
1
1
Mitigation for XS 6.2 Use an event-channel for buffered I/O notifications
Provide options to disable all emulated devices
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
28 / 34

29. Benchmarks
Outline
1
Scalability expectations
2
Hard limits
3
Soft limits
4
Benchmarks
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
29 / 34

30. Benchmarks
Bootstorm
Booting 90 Win7 VMs
Time to fully boot 90 VMs (25 at a time)
2500
Tampa (XS 6.1)
Clearwater (XS 6.2)
Elapsed time (s)
2000
1500
1000
XS 6.2 is
60% faster
500
0
0
25
50
75
100
125
150
175
200
VM index
Machine used: Dell PowerEdge R815 (Quad 16-core AMD Opteron 6272 @ 2.1GHz)
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
30 / 34

31. Benchmarks
Bootstorm
Booting 120 Win7 VMs
Time to fully boot 120 VMs (25 at a time)
2500
Tampa (XS 6.1)
Clearwater (XS 6.2)
Elapsed time (s)
2000
1500
1000
XS 6.2 is
75% faster
500
0
0
25
50
75
100
125
150
175
200
VM index
Machine used: Dell PowerEdge R815 (Quad 16-core AMD Opteron 6272 @ 2.1GHz)
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
31 / 34

32. Benchmarks
Bootstorm
Booting 200 Win7 VMs
Time to fully boot 200 VMs (25 at a time)
2500
Tampa (XS 6.1)
Clearwater (XS 6.2)
XS 6.1 can't
even get 200
VMs running!
Elapsed time (s)
2000
1500
It took XS 6.2 just
13 minutes to boot
200 VMs
(on this hardware)
1000
500
0
0
25
50
75
100
125
150
175
200
VM index
Machine used: Dell PowerEdge R815 (Quad 16-core AMD Opteron 6272 @ 2.1GHz)
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
32 / 34

33. Benchmarks
LoginVSI
number of VMs performing acceptably
LoginVSI: number of usable Windows VMs
XS 6.2
XS 6.1
number of VMs running
Machine used: Quad 10-core Intel E7-4860 @ 2.27GHz
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
33 / 34

34. Questions
Questions
?
Jonathan Davies (Citrix)
Increasing XenServer’s VM density
24 Oct 2013
34 / 34