2. • Traditional tradeoff:
Local vs. Central storage:
– Central storage migrates very simply.
• Just complete last work from old location
– Then start at the new location.
• Extracts non-local latency tax on every access.
– Purely local storage makes migration a challenge.
• Must migrate VM image and local storage.
• And synchronize them.
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/Alex.Aizman@nexenta.com 2
3. • SSD vs HDD:
–SSDs provide most IOPS per $.
• SSDs need to be on host, or at
least near the VM, or network
latencies minimize SSD speed.
–HDDs provide most TBs per $.
• HDDs provide capacity, not
processing speeds.
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/Alex.Aizman@nexenta.com
4. • A very common solution currently features Host
DomU
a Block Backend that co-ordinates a local App
block device (typically SSD) with remote VFS
File System
block device (still NAS or SAN). Block
Driver
• Problem: Device Domain
– Any cache solution only stores a portion Block Backend SSD Proxy
of a VM‟s storage locally. Remote Block Local Block
Driver
Access
– With NAS Proxy the “portion stored
locally” can be understood as files and File System
Block
NAS Client
directories. Network
Driver
Network
• A block-oriented solution never sees Driver Driver
more than a collection of blocks. Local Block
Device
• Many solutions are enhanced or External
NAS
External
SAN
made possible with File-orientation. Server(s) Server(s)
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/Alex.Aizman@nexenta.com 4
5. Host
• DomU can support both NAS DomU App
access and Virtual Disk Access VFS
– Most VMs will prefer directly-attached boot File NAS Client
System
drive. Block Network
Driver Driver
• Backend can map virtual drive to Device Domain
Block Network
NAS. Backend Backend
• Meanwhile NAS Client can access
NAS Proxy
Network Block
NAS Proxy via local internal Driver Driver
network. Local Block
Device
• NAS Proxy can utilize local SSD External
NAS
Server(s)
drive to optimize either.
5 Xen Summit Aug 2, 2011 –
Caitlin.Bestler/Alex.Aizman@nexenta.com
6. • NAS Benefits
– Files enable understanding of context:
– Deduplication of files is far easier than deduplication of block.
– Hierarchical Storage Management works on files, not blocks.
• Traditional NAS Proxy Caveats
– Traditional NAS is unaware of VM-specific semantics.
– Excessive syncing – Historical assumptions are that files are
shared by default.
– Proxy overhead for non-cached files – Historically, the only easy
configuration was to proxy all NAS access.
• VM Storage Proxy provides full benefits of NAS without
limitations of traditional NAS Proxies.
6 Xen Summit Aug 2, 2011 –
Caitlin.Bestler/Alex.Aizman@nexenta.com
7.
8. • Distributed NAS protocols such as NFSv4 and
DFS/CIFS have features that make NAS proxies far
more efficient.
• They allow efficient caching via exclusive delegation
and oplocking (“opportunistic locking”).
– Proxy can temporarily claim a file.
• They have advanced session features to protect
against data loss from connection loss.
– File reverts to central NAS server if Proxy dies.
– System recovers from network partitions and temporary
outages of distributed servers and/or proxies as long as
they all have persistent redundant storage.
Xen Summit Aug 2, 2011 –
8 Caitlin.Bestler/Alex.Aizman@nexenta.com
9. • Client can simply mount from NAS
server.
VM Client
• NAS server can refer select
directories to a NAS “Server” that
is actually the NAS Proxy. NAS Proxy /
Server
– NAS Proxy/Server must act as Referral
a NAS Server for NAS Replication
replication procedures. Cached
Files
• Client does not need to identify Directory access
itself.
– NAS server can infer identity
from IP address. All
NAS Server
Directories
• Advantage that Proxy/Server is
enabled/disabled for each directory
from the central NAS Server.
Xen Summit Aug 2, 2011 –
9 Caitlin.Bestler/Alex.Aizman@nexenta.com
10. • NAS Proxy can act as a NAS
VM Client
client and claim an exclusive
delegation on a file.
• It can then treat its cached copy Local copy can
be treated as
THE copy.
as the copy.
Cached NAS Proxy /
• It must replicate it back to the Files Server
central NAS server before
Delegation
ending the delegation. Granted
Claim Exclusive Delegation
• Keep-alive messages track the
NAS client with the delegation. All
NAS Server
Directories
• Protocol rules already cover
loss of contact/failure
scenarios.
Xen Summit Aug 2, 2011 –
10 Caitlin.Bestler/Alex.Aizman@nexenta.com
11.
12. • Generic NAS Proxy does not understand that it is the sole
writer for the VM Private Directory.
• NAS Proxy management is not integrated with VM
Management.
– The NAS system must detect that the VM using its VM Private
Directory has migrated.
• Proxies only optimize storage for selective files – but all
traffic goes through them.
– If there are directories/files that are not worth caching the clients
would ideally just go directly to the real server.
– NAS Referrals can be applied selectively, but Management Plane
controls are awkward for dynamically controlling referrals.
• Optimized cache sharing across multiple clients is
challenging without understanding of VM life-cycle.
Xen Summit Aug 2, 2011 –
12 Caitlin.Bestler/Alex.Aizman@nexenta.com
13. • Allow VM‟s files to be stored and archived as
files.
• Allow storage associated with commodity servers
to be deployed effectively for VMs in combination
with NAS storage.
• Don‟t replicate to local storage those files that
probably won‟t be accessed anyway.
• Allow very thin provisioning of local storage.
– SSDs are designed to provide IOPs cost effectively,
not storage.
– If IOPS are not needed, avoid cost of redundant
storage.
Xen Summit Aug 2, 2011 –
13 Caitlin.Bestler/Alex.Aizman@nexenta.com
14. • The VM NAS Proxy is a specialized NAS Proxy that is optimized to
support VMs and the Hypervisor Platform supporting them:
• A VM Private Directory is maintained for each VM:
– A per VM directory in the NAS global name space that
holds files that are “local” for this VM.
– A “platform” sub-directory hold virtual disk images and
other files used by the Hypervisor and/or Dom0.
– A “mount” sub-directory holds NAS mount points that
are available only to this VM.
• These are implicitly named. The client does not know the
path to its VM Private Directory because it does not know
which VM it is.
• A “proxy” sub-directory holds files used by the VM NAS Proxy
for this VM, such as the NAS Response Cache.
Xen Summit Aug 2, 2011 –
14
Caitlin.Bestler/Alex.Aizman@nexenta.com
15. VM NAS Proxy as a VM. VM NAS Proxy for a Rack
Virtualization Host Rack (or other enclosure)
Virtualization Host
Hypervisor Platform VM NAS Proxy
VM13478 VM
Hypervisor Platform VM13478
VM
/foo VM NAS Proxy
/foo
/bar
/bar
NAS VMS
/vmexports /vmexports
/vm13478 NAS VMS /vm13478
/platform /platform
/mounts /mounts
NAS NAS
/foo /foo
` /bar ` /bar
/vm13483 /vm13483
/platform Virtualization Management
Virtualization Management /platform
/mounts /mounts
/foo /foo
/bar /bar
15 Xen Summit Aug 2, 2011 –
Caitlin.Bestler/Alex.Aizman@nexenta.com
16. • Virtualization Management
– Unchanged, but new information is available to it.
• NAS VMS (Virtualization Management System)
– Extends Virtualization Management to assign VM NAS Proxies.
• Hypervisor Platform
– The Hypervisor and supporting domains.
• NAS system with NFSv4/v4.1 and/or DFS/CIFS.
– Unchanged, but optional enhancements are defined.
• VMs
– Unchanged, but optional enhancements for NAS Client are
defined.
• VM NAS Proxy
– A specialized NAS Proxy optimized for support of VMs
Xen Summit Aug 2, 2011 –
16
Caitlin.Bestler/Alex.Aizman@nexenta.com
17. • A VM NAS Proxy can control what is
stored locally through several techniques:
– Which sub-directories of the VM‟s private
directory are referred to it.
– Whether it provides local storage for specific
files within those directories.
– Which blocks of a file that is stored locally are
actually stored locally, vs. being accessed
over the network.
– pNFS-aware clients can be enabled to access
network block servers directly.
Xen Summit Aug 2, 2011 –
17
Caitlin.Bestler/Alex.Aizman@nexenta.com
18.
19. • VM will be migrated from Current Host Virtualization Host
Current Home
to a new Host.
10.1.3.115Hypervisor Platform
VM
• VM has public IP address, which 192.168.1.115
10.1.1.1
192.168.1.1
Virtualization Management will migrate. VM NAS Proxy
10.1.1.2
• VM has private IP Address which it will
192.168.1.2
use in new network without realizing that
it is on a new network. Planned Home
• New VM NAS Proxy will have same Virtualization Host
private IP address on the new network. 10.1.3.115
Hypervisor Platform
10.1.2.1
192.168.1.1 NAS
192.168.1.115
• New VM NAS Proxy does not respond
10.1.2.2
192.168.1.2
to the VM‟s private IP Address until it is
ready. VM Storage Proxy
Xen Summit Aug 2, 2011 –
19 Caitlin.Bestler/Alex.Aizman@nexenta.com
20. • By assigning VM NAS Proxies to an Internal Network
migration of a VM can be made transparent to the VM.
– All Internal Networks can look alike.
– Just as almost all home networks are 192.168.1.*
• NAS client does not need to know about the migration.
– NAS clients will see a dropped connection to a NAS server.
– NAS clients will assume the NAS server is simply rebooting.
– Recovery will be complete and transparent to client layers.
– No change to the IP address of the VM NAS Proxy the Client
uses.
• As far as the client is concerned it just reconnected to
the same server
– It even has the same response cache.
– The response cache is stored within the VM Private Directory.
20 Xen Summit Aug 2, 2011 –
Caitlin.Bestler/Alex.Aizman@nexenta.com
21. • VM will be migrated from Current
Current Home
Virtualization Host
Host to a new Host.
– 10.1.3.115 is migrated.
VM
Hypervisor Platform
10.1.3.115
10.1.1.1
• VM is still instructed to use the old
VM NAS Proxy (10.1.1.2). 10.1.1.2 VM NAS Proxy
– It will continue to use it until explicitly
redirected to the new VM NAS Proxy ac
ce
ss
d
ue
(10.1.2.2). Co
nt
in
Planned Home
– This is very network inefficient. Virtualization Host
– Current RTT measurements and
Hypervisor Platform
10.1.3.115
congestion window will be wildly 10.1.2.1
NAS
incorrect. 10.1.2.2
– VM NAS Proxy must accept clients
from anywhere.
VMNAS Proxy
Xen Summit Aug 2, 2011 –
21 Caitlin.Bestler/Alex.Aizman@nexenta.com
22.
23. • Distributed NAS features for Referral and
Delegation have nothing to do with each other.
– Referral/redirection is a Management Plane
feature.
• It is used to refer/redirect clients to other NAS
servers.
– Delegation/oplocking is a Control Plane feature.
• It is used by NAS clients.
• A VM NAS Proxy integrates them in
collaboration with NAS servers.
Xen Summit Aug 2, 2011 –
23 Caitlin.Bestler/Alex.Aizman@nexenta.com
24. • VM NAS Proxies use Delegation/Oplocking to
exclusively claim specific sub-directories
within a VM Private Directory.
– This is an enhancement to a NAS server.
• Supporting NAS servers will then
automatically direct referrals for the claimed
sub-directories to that VM NAS Proxy for the
duration of the delegation/oplock.
– Reverting is automatic, preferable to having to
reverse a management plane edit.
Xen Summit Aug 2, 2011 –
24 Caitlin.Bestler/Alex.Aizman@nexenta.com
25. • Only the correct clients are referred,
• Other clients are refused access.
• Allowed access types:
– Hypervisor Platform access a „platform‟ sub-
directory for virtual disk images and snapshots.
– VM itself accesses NAS mounts in a „mount‟ sub-
directory.
– VM NAS Proxy keeps its own data in a „proxy‟
sub-directory.
• This is where the NAS Response Cache is kept.
Xen Summit Aug 2, 2011 –
25 Caitlin.Bestler/Alex.Aizman@nexenta.com
26. 2
3 1
VM NAS Proxy
NAS
NAS
Server
Server
Locally
Cached
Files
• 1 – VM NAS Proxy takes exclusive delegation on a directory.
• 2 – Client referencing delegated directory is referred to VM NAS
Proxy
• 3 – Client accesses directory on VM NAS Proxy
Xen Summit Aug 2, 2011 –
26 Caitlin.Bestler/Alex.Aizman@nexenta.com
27. • Progression of Solutions for Controlling File
Locations:
– Without NFSv4 Referrals/DFS Redirections:
• Load balancing requires editing each client‟s configuration.
• Or directing all traffic through a proxy.
– Using just standard Referrals/Redirections:
• Central control of which directories are referred
• But only via Management Plane edits of exports.
– Integrating Delegation to allow Automatic Referrals:
• Dynamic control over which directories are referred is
exercised by the VM NAS Proxy and/or VM NAS
Management.
• Dynamic referral automatic ends when delegation ends.
Xen Summit Aug 2, 2011 –
27 Caitlin.Bestler/Alex.Aizman@nexenta.com
28.
29. • One of the features of the VM NAS Proxy is that it can
dynamically control which of a VM‟s private directories
are referred to it.
– But standard granularity for referral is an entire directory.
• Partial Referrals is a new technique to publish alternate
referrals for subsets of a directory:
– Each partial referral identified by a regular expression.
– The alternate server is then specified for that regular expression,
which can be “revert to default” to ignore the main referral.
– This information is published as named attributes for the
directory.
• If a NAS client does not understand them then it will simply
go to the default server for the directory
• The default server then acts as a proxy for the actual server.
Xen Summit Aug 2, 2011 –
29 Caitlin.Bestler/Alex.Aizman@nexenta.com
30. 2B
1
2A
VM NAS Proxy
NAS
NAS
Server
Server
Locally
Cached
Files
• 1 – VM NAS Proxy advertises portions of directory that Client should access directly
on the home NAS server.
• 2a – Client continues to access non-excluded portions of directory on the VM NAS
Proxy, which caches them locally.
• 2b – But now accesses files covered by partial referral on the home NAS Server
Xen Summit Aug 2, 2011 –
30 Caitlin.Bestler/Alex.Aizman@nexenta.com
31. 1
2 3
VM NAS Proxy
NAS
NAS
Server
Server
Locally
Cached
Files
• 1 – VM NAS Proxy advertises portions of directory that Client should access
directly on the home NAS server.
• 2 – Client does not understand or ignores the partial referral, so all access for the
directory goes through the VM NAS Proxy
• 3 – When files are not stored locally access is proxied via the NAS Server.
Xen Summit Aug 2, 2011 –
31
Caitlin.Bestler/Alex.Aizman@nexenta.com
32.
33. • When the VM‟s NAS Client
indicates that it understands
pNFS, the VM NAS Proxy Locally Cached Blocks
can supply pNFS layouts Logical Physical
Block # Block #
that mix references to its Logical Physical
Proxy pNFS Layout
own storage and NAS Block # Block # Logical
Block #
Local
Volume ID
Physical
Block #
storage. Logical Physical
•
Volume ID
Referencing pNFS blocks Block # Block #
Logical Physical
directly is safe, because the Block #
Volume ID
Block #
VM NAS Proxy is the only pNFS Layout Logical Local Physical
client allowed to update the Logical Physical
Block # Volume ID Block #
Volume ID Logical Physical
file while it is claimed. Block # Block #
Block #
Volume ID
Block #
• pNFS-aware clients can Logical
Block #
Volume ID
Physical
Block #
Logical
Block #
Volume ID
Physical
Block #
access unmodified and Logical
Volume ID
Physical Logical Physical
Block # Block # Volume ID
uncached blocks without Logical Physical
Block # Block #
overhead of going through Block #
Volume ID
Block #
the proxy.
Xen Summit Aug 2, 2011 –
33
Caitlin.Bestler/Alex.Aizman@nexenta.com
34. • Named attributes can identify files that are
to be stored only in the local cache.
– The file is never updated on the NAS server.
• Regular expression can easily identify
common temporary files.
• These files would be stored only on local
SDD
– never consuming any network bandwidth.
Xen Summit Aug 2, 2011 –
34
Caitlin.Bestler/Alex.Aizman@nexenta.com
35.
36. • VM NAS Proxy is only
on the path for a VM pNFS Block
Server
accessing storage Direct access to un-cached blocks
when it wants to be. VM
– Control is fully dynamic.
Default Access, may be referred to VM Storage Proxy
Access to non-referred directories
Access to partially un-referred files
• Access to Non- NAS
Server
optimized Access to referred directories, files
directories/files can
VM NAS PRoxy
bypass the Proxy.
• VM NAS Proxy is only
identified as referral
target for the correct
clients.
Xen Summit Aug 2, 2011 –
36 Caitlin.Bestler/Alex.Aizman@nexenta.com