Adding multitenant storage capability to the illumos kernel

1. Adding Multitenant Storage
Capability to the illumos Kernel
Richard Elling and Garrett D’Amore

2. Agenda
• Why multitenancy?
• Survey of available technologies
• OS virtualization
• Multitenant storage services in illumos
February 24 2012 Presented at the SCALE 11x Conference

3. Why Multitenancy?
February 24 2012 Presented at the SCALE 11x Conference

4. Userland Process Userland Process
Kernel Device Driver
Simple model: multitasking OS
February 24 2012 Presented at the SCALE 11x Conference

5. Strike up the band!
February 24 2012 Presented at the SCALE 11x Conference

6. Guest
Userland Process
Kernel Device Driver
Hypervisor Device Driver
Native Hypervisor Model
Examples: VMware ESX, XenServer, KVM
February 24 2012 Presented at the SCALE 11x Conference

7. Java
Java Application
JVM APIs
OS Device Driver
Java Virtual Machines are Virtual Machines, too
February 24 2012 Presented at the SCALE 11x Conference

8. Guest
Userland Process
Userland Kernel Device Driver
Process
Hypervisor Device Driver
Kernel Device Driver
Hosted Hypervisor Model
Examples: VMware Workstation, VirtualBox
February 24 2012 Presented at the SCALE 11x Conference

9. Userland Process
HVM
Guest I/O Proxy
OS Device Driver
QEMU
Host
OS
Device Driver
KVM
February 24 2012 Presented at the SCALE 11x Conference

10. Userland Process
DomU Device Driver
Front-end driver
Back-end driver
Dom0 Userland I/O Proxy
Device Driver
XenServer
February 24 2012 Presented at the SCALE 11x Conference

11. Userland Process Userland Process
Kernel Device Driver
Return to simple model: multitasking OS
February 24 2012 Presented at the SCALE 11x Conference

12. Zone Zone
Userland Process Userland Process
Device Driver
Kernel
February 24 2012 Presented at the SCALE 11x Conference

13. Jail
Userland Process
Directory
structure IP Address
BSD
File system
Kernel
Storage Device Drivers NIC Driver
BSD Jails
February 24 2012 Presented at the SCALE 11x Conference

14. Zone
Userland Process
TCP/IP
File system
Virtual NIC
Kernel
Classification Engine
Storage Device Drivers NIC Driver
illumos OS
February 24 2012 Presented at the SCALE 11x Conference

15. The first thing we do, let’s kill all the lawyers
King Henry VI
Act IV, Scene II
February 24 2012 Presented at the SCALE 11x Conference

16. The first thing we do, let’s kill all the lawyers
King Henry VI
Act IV, Scene II
The first thing we do, let’s kill all the layers
UNIX Performance Gurus
February 24 2012 Presented at the SCALE 11x Conference

17. Zone Zone
Userland Process Userland Process
Device Driver
Kernel
February 24 2012 Presented at the SCALE 11x Conference

18. Kernel
Zone A
Zone B
NFS SMB NFS
iSCSI TCP/IP TCP/IP
File system File system File system
Block device Virtual NIC Virtual NIC
Transactional Object Store Virtual
and Data Management Unit Switch
Storage Device Drivers NIC Driver
February 24 2012 Presented at the SCALE 11x Conference

19. Global state ➾ Per-zone state
NFS - rpc now zoned
SMB - in-kernel, multithreaded, zoned
iSCSI - per-zone block devices
February 24 2012 Presented at the SCALE 11x Conference

20. Zone VM
NFS NFS
FS Cache
Kernel Zone kernel Traditional
SMB
Hypervisor
Zone VM
iSCSI Samba
FS Cache
NFS
kernel
Adaptive Replacement
Cache
Big shared caches Inefficient small caches
~ 100 MB/tenant > 1000 MB/tenant
February 24 2012 Presented at the SCALE 11x Conference

21. Kernel
Zone A
Zone B
ANP iSCSI iSCSI TCP/IP TCP/IP
New
Block device
Awesome interface? Virtual NIC Virtual NIC
New Block
Device device
Transactional Object Store Virtual
and Data Management Unit Switch
Storage Device Drivers NIC Driver
Deep access to devices
February 24 2012 Presented at the SCALE 11x Conference

22. Zone Zone
NFS iSCSI NFS SMB
TCP TCP
IP Multipath Group IP Multipath Group
Virtual NIC Virtual NIC Virtual NIC Virtual NIC
Virtual Switch Virtual Switch
Physical NIC Physical NIC
Virtual Networking
February 24 2012 Presented at the SCALE 11x Conference

23. Zone A
Resource caps
Network usage
I/O usage
Zone B
All kstats
Resource caps
Network usage
I/O usage
Kernel statistics ✦ kstats
February 24 2012 Presented at the SCALE 11x Conference

24. All in due time?
February 24 2012 Presented at the SCALE 11x Conference

25. Tenant A Tree AB
Domain A
Tenant B
A SMB
B SMB Domain B
C NFS ✦ SMB
Tenant C
Domain C
February 24 2012 Presented at the SCALE 11x Conference

26. Summary
• Efficient, multitenant storage in illumos
• Sit out of the traditional VM dance
• OS virtualization rulez!
February 24 2012 Presented at the SCALE 11x Conference

27. www.illumos.org
www.dey-sys.com
www.RichardElling.com
Richard.Elling@dey-sys.com
Garrett.D’Amore@dey-sys.com
February 24 2012 Presented at the SCALE 11x Conference

28. Adding Multitenant Storage
Capability to the illumos Kernel
Richard.Elling@dey-sys.com
Garrett.D’Amore@dey-sys.com