Samuel Thibault: Stub Domains

1. Stub Domains
A Step Towards Dom0 Disaggregation
Samuel Thibault, Citrix/XenSource

2. The Big Domain 0
Runs a lot of Xen components

Domain manager
◦
Domain Builder
◦
Device Models
◦
PyGRUB
◦
These are currently running as root

◦ e.g. PyGRUB to access guest's disk
Security issues

Scalability issues


3. What Are Stub Domains?
Helper domains which run Xen components

Based on Mini-OS

Domain Builder (Derek Murray)

Device Model

PV-GRUB

...


4. What Are Stub Domains?
Helper domains which run Xen components

Based on Mini-OS

Domain Builder (Derek Murray)

Device Model

PV-GRUB

...


5. POSIX Environment
on Top of Mini-OS
A p p l i c a ti o n
getpid,
sleep,
n e w lib lw IP
U n ix
read, lines
1200
select, ..
C o n s o le B lo c k FS FB
N e tw o rk
Sched MM fro n te n d fro n te n d fro n te n d fro n te n d
fro n te n d
M in i- O S
X e n H y p e r v is o r

6. New Mini-OS Features
Disk frontend

FrameBuffer frontend

FileSystem frontend

◦ Imported from JavaGuest
◦ Remote access to some /export (e.g. of dom0)
More advanced MM

◦ Read-Only memory
◦ CoW for zeroed pages
But still keep it simple

◦ Single address space, mono-VCPU, no preemption
Bugfixes!


7. stubdom/
 Makefile
◦ Download and compile a cross-compilation
environment
 binutils, gcc, newlib, lwip
 c/
◦ 'Hello World!' C application
 caml/
◦ 'Hello World!' Caml application
 README
◦ Of course :)

8. Current HVM device model
qem u
L in u x
H V M d o m a in
dom 0
IN /O U T
X e n H y p e rv is o r

9. Current HVM dm
Not always responsive

◦ Have to wait for dom0 Linux to schedule qemu
Eats dom0 CPU time

Uses dom0 resources from userland

◦ Disk, tap network
◦ Hence runs as root

10. HVM dm domain
qem u
PV
L in u x M in i-O S
H V M d o m a in
dom 0 s tu b d o m
IN /O U T
X e n H y p e rv is o r

11. HVM dm domain
45
40
35
30
25
Dom0
20 Stubdom
15
10
5
0
Inb (Kcy) Boot time (s)

12. HVM dm domain Disk Perfs
80
70
60
50
Dom0
40 Stubdom
30 Native
20
10
0
Read (MB/s) Write (MB/s)

13. HVM dm domain Disk CPU%
stubdom
DomU
Dom0
Read dom0 Stubdom
Free
stubdom
DomU
Dom0
Write dom0 Stubdom
Free

14. HVM dm domain Net Perfs
e1000
80
70
60
50
Dom0
40
Stubdom
30
20
10
0
Recv (MB/s) Send (MB/s)

15. HVM dm domain Net CPU%
e1000 stubdom
DomU
Recv Dom0
dom0 Stubdom
Free
stubdom
DomU
Send Dom0
dom0 Stubdom
Free

16. HVM dm domain Net Perfs
bicore
120
100
80
Dom0
60
Stubdom
40
20
0
Recv (MB/s) Send (MB/s)

17. HVM dm domain Net CPU%
bicore stubdom
DomU
Recv Dom0
dom0 Stubdom
Free
stubdom
DomU
Send Dom0
dom0 Stubdom
Free

18. HVM dm domain
Almost unmodified qemu

◦ Disable e.g. sound support, plug Mini-OS PV drivers
Relieves dom0

Provides better CPU usage accounting

◦ Can charge HVM domain with dm domain time
A lot safer

◦ Only privilege is having the HVM dom as target
◦ Uses same resource access as PV guests
More efficient

◦ Let the hypervisor schedule it directly
◦ More lightweight OS

19. PyGRUB
PyG R U B
xend
L in u x
P V d o m a in
dom 0
X e n H y p e rv is o r
menu.lst
vmlinuz
initrd

20. PyGRUB
Needs to be root to access guest disk

◦ Security issues
Does not currently provide network boot

Reimplements GRUB


21. PV-GRUB start
xend
GRUB lib x c
L in u x
M in i-O S
dom 0
X e n H y p e rv is o r
menu.lst
vmlinuz
initrd

22. PV-GRUB loading
xend P V k e rn e l in itrd
GRUB lib x c
L in u x
M in i-O S
dom 0 b lk f r o n t n e tfr o n t
X e n H y p e rv is o r
menu.lst
vmlinuz
initrd

23. PV-GRUB loaded
xend P V k e rn e l in itrd
GRUB lib x c
L in u x Kexec!
M in i-O S
dom 0
X e n H y p e rv is o r

24. PV-GRUB
xend P V k e rn e l in itrd
L in u x
P V d o m a in
dom 0
X e n H y p e rv is o r

25. PV-kexec
in i t r d
P V k ern el
boot
kexec
li b x c
GRUB
M in i- O S
M in i- O S
v ir t u a l m e m o r y

26. PV-kexec
s ta c k
in it r d
P V k ern el p g t a b le
in it r d
boot
P V k ern el
kexec
0xc0000000
lib x c
GRUB
M in i- O S
M in i- O S T arg et P V g u est
v ir t u a l m e m o r y v ir t u a l m e m o r y

27. PV-kexec
s ta c k
in it r d
boot
P V k ern el p g t a b le
in it r d
boot
P V k ern el
kexec
0xc0000000
lib x c
GRUB
M in i- O S
M in i- O S T arg et P V g u est
v ir t u a l m e m o r y v ir t u a l m e m o r y

28. PV-kexec
s ta c k
in it r d
boot boot
P V k ern el p g t a b le
in it r d
boot
P V k ern el
kexec
0xc0000000
lib x c
GRUB
M in i- O S
M in i- O S T arg et P V g u est
v ir t u a l m e m o r y v ir t u a l m e m o r y

29. PV-kexec
s ta c k
in it r d
boot boot
P V k ern el p g t a b le
in it r d
boot
P V k ern el
kexec
0xc0000000
lib x c
GRUB
M in i- O S
M in i- O S T arg et P V g u est
v ir t u a l m e m o r y v ir t u a l m e m o r y

30. PV-kexec
s ta c k
in it r d
boot boot
P V k ern el p g t a b le
in it r d
boot
P V k ern el
kexec
0xc0000000
lib x c
GRUB
M in i- O S
M in i- O S T arg et P V g u est
v ir t u a l m e m o r y v ir t u a l m e m o r y

31. PV-kexec
s ta c k
in it r d
boot boot
P V k ern el p g t a b le
in it r d
boot
P V k ern el
kexec
0xc0000000
lib x c
GRUB
M in i- O S
M in i- O S T arg et P V g u est
v ir t u a l m e m o r y v ir t u a l m e m o r y

32. PV-GRUB
Executes upstream GRUB

◦ Replace native drivers with Mini-OS drivers
◦ Add PV kexec implementation
Just uses the target PV guest resources

Supports network

Supports graphical menu


33. Conclusion
Dm domain

Improves security
◦
Improves accounting
◦
Improves scalability
◦
Improves performances
◦
PV-GRUB

◦ Improves security
◦ Provides network boot
Mini-OS also being tested at Cisco for IOS

Available in the unstable tree


34. Future Work
Dm domain

◦ Live migration, PCI PT
◦ IA-64 support
◦ Group scheduling with HVM domain
PV-GRUB

◦ Kexec 64bit guest from 32bit PV-GRUB
◦ PVFB shutdown/restart
OCaml support

◦ 'Hello World!' works
◦ Needs runtime rebuild to properly hook into POSIX
layer