We’ve pushed the backup performance envelope so that you don’t have to! Industry leaders Cisco, NetApp, VMware and Symantec teamed up to develop a best practice framework and performance benchmark based on the vStorage APIs for Data Protection (VADP). The test configuration uses the popular NetApp FlexPod environment. The result that proves that you can easily protect over 4 TB of virtual machine data an hour. Most think that to provide this sort of performance, a mountain of hardware is required. This is not the case. We show you how these performance numbers can be easily obtained with minimal hardware and a small budget. Improving backup performance also creates more reliable backups, shorter backup windows and less impact on the vSphere infrastructure. Whether or not you are using NetBackup, we invite you to attend this hangout where we'll dig deep into VMware VADP and its performance characteristics. We'll share lessons learned from extensive lab benchmarks simulating real production workloads. This will help you design and deploy a backup solution for your VMware vSphere environment that meets your business SLAs. Bonus exclusively at this hangout: In this hangout, we will provide a sneak preview some of NetBackup for VMware features coming in NetBackup 7.6. We have exciting results to share on how NetBackup is pushing the performance envelope further! Panel Members: • Abdul Rasheed, Product Marketing Manager, vExpert • George Winter, Technical Product Manager, vExpert • Alex Sakaguchi, Product Marketing Manager View Hangout: http://bit.ly/1efz3zB

1. Disclaimer
This information is about pre-release software. Any unreleased
update to the product or other planned modification is subject
to ongoing evaluation by Symantec and therefore subject to
change.
This information is provided without warranty of any kind,
express or implied. Customers who purchase Symantec
products should make their purchase decision based upon
features that are currently available.
1

2. Participate in the discussion!
2
#SYMCHangout
http://bit.ly/1efz3zB

3. VMware Backups That Work
George Winter Technical Product Manager, Symantec
Abdul Rasheed Technical Marketing Manager, Symantec
Lessons Learned From VADP Performance Benchmark Testing
3

4. Agenda – NOTE: UPDATE THIS
Why Perform These Benchmarks?
Benchmark Testing History
Benchmark – NBD Transport
Benchmark – SAN Transport
VADP Process – Quick Review
The backup snapshot process
HW and Backup Configuration Recommendations
Which Method is Fastest?
4

5. Benchmark Testing History
5

6. Three VMware Backup Benchmarks
• Requires significant hardware & very slow
• VCB has since been deprecated (not going to discuss)
VMware Consolidated Backup (VCB)
• vStorage API for Data Protection
• NetApp FlexPod, Cisco UCS, NetBackup
NBD (network) benchmark
• vStorage API for Data Protection
• Cisco UCS, NetBackup
SAN transport benchmark
6

7. Benchmark Environments – Real World Configurations
7
Cisco UCS Blades
1x vCenter, 6x ESXi
Nexus
NetBackup 5220
NetApp FAS 6080 NetApp V-Series 6080
Block Storage
8Gb Fibre Channel
10Gb Ethernet

8. Why Perform These Benchmarks?
Benchmarks Are A Lot Of Work!
8

9. Benchmark Goal – Better Backups For You!
Trust but verify – never assume performance
Develop independent performance testing techniques
Tell You What We Learned
Understand hardware and backup API (VADP) limitations
Faster backups
More reliable
Correctly size backup server
Shorter backup window
Restore performance
Restore options
9

10. 10
VADP Backup Process
Quick Review

11. VADP Backup Process
11
VMware ESXi
Datastore
Backup
Server
ESXi Host

12. VADP Backup Process
12
VMware ESXi
Datastore
Backup
Server
VADP Snapshot
of Target VM(s)
ESXi Host

13. VADP Backup Process
13
VMware ESXi
Datastore
Backup
Server
VM Backup
Sent to Server
ESXi Host

14. VADP Backup Process
14
VMware ESXi
Datastore
Backup
Server
Snapshot
Released
ESXi Host

15. Participate in the discussion!
15
#SYMCHangout
http://bit.ly/1efz3zB

16. 16
Backup Snapshots Exposed!
Why Do You Care?

17. A Closer Look - What Happens During a Backup?
• VSS provider flushes OS buffers in VM
• Snapshot of VM is taken (vmdk(s) are frozen)
(SCSI reservation of LUN)
• Redo log created – all writes redirected to redo log
(Redo expanded in 16 MB chunks)
• VM is backed up
• Redo log data applied to original vmdk(s)
• Snapshot released - backup completed
During VM backups, VMware creates temp
snapshot of VM. Process is as follows:
• Every one of these steps involves significant amount of I/O
• Reducing number of snapshots per Datastore improves backup perf & reliability
• Incrementals are quick and can reduce snapshot impact
Why does this matter?
A Detailed Look at the VADP Snapshot Process
17
VMware ESXi

18. Reducing Backup I/O Impact – Improve B/U Reliability
• Backup during periods of low VM activity
• Use incremental backups (CBT) liberally
• Limit simultaneous backups per ESX / Datastore (VIP)
Shorten time backup snapshot is open
• Design backup policies to evenly balance load across ESXi / Datastores
• Faster backups = shorter time snapshots are open
Configure NetBackup for optimal performance
• Snapshots more reliable
• Overall backup processing faster
Result:
18

19. 19
VADP Snapshot Pop Quiz

20. VIP Configuration Exercise – NBD Transport
20
UCS
VMDKVMDK
VM2
ESXi
ESXi
ESXi
ESXi
ESXi
ESXi
Network
Backup
Server
• Network = 10GbE
• 6 ESXi Hosts
• VMs = 100 GB
Q: How many streams per ESXi host will
provide fastest possible backup?
1, 2, 3, 4, more?
10GbE

21. 1 Stream / ESXi Host
(6 streams total)
Network = 10 GbE
100 MB/sec per ESXi
Each VM = 100 MB/sec
Backup 21 VMs / Hour
NBD Transport – How Many Streams?
21
UCS
VMDKVMDK
VM2
ESXi
ESXi
ESXi
ESXi
ESXi
ESXi
3 Streams / ESXi Host
(18 streams total)
Network = 10GbE
100 MB/sec per ESXi
Each VM = 33 MB/sec
Backup 21 VMs / Hour

22. VIP Configuration Example – NBD Transport (Review)
22
UCS
VMDKVMDK
VM2
ESXi
ESXi
ESXi
ESXi
ESXi
ESXi
Network
Backup
Server
• Network = 10GbE
• 6 ESXi Hosts
• VMs = 100 GB
10GbE

23. NBD Transport – Impact Of Snapshots Per Datastore?
23
UCS
VMDKVMDK
VM2
ESXi
ESXi
ESXi
ESXi
ESXi
ESXi
3 Streams / ESXi Host
(18 streams total)
Snapshots / DS = 3
Redo Space Req’d:
45 GB
Per Datastore
1 Stream / ESXi Host
(6 streams total)
Snapshot / DS = 1
Redo Space Req’d:
5 GB
Per Datastore
???

24. NBD Transport – Impact Of Snapshots Per Datastore?
24
UCS
VMDKVMDK
VM2
ESXi
ESXi
ESXi
ESXi
ESXi
ESXi
3 Streams / ESXi Host
(18 streams total)
Snapshots / DS = 3
Redo Space Req’d:
45 GB
Per Datastore
1 Stream / ESXi Host
(6 streams total)
Snapshot / DS = 1
Redo Space Req’d:
5 GB
Per Datastore
45 GB

25. NBD Transport – Impact Of Snapshots Per Datastore?
25
UCS
VMDKVMDK
VM2
ESXi
ESXi
ESXi
ESXi
ESXi
ESXi
3 Streams / ESXi Host
(18 streams total)
Snapshots / DS = 3
Redo Space Req’d:
45 GB
Per Datastore
1 Stream / ESXi Host
(6 streams total)
Snapshot / DS = 1
Redo Space Req’d:
5 GB
Per Datastore
Why 45 GB for 3 simultaneous backups?
• Each VM backs up at slower rate
• Snapshot for each VM is open 3 times as long
• Time to capture data change in redo is longer
• Does not improve overall backup performance!!
45 GB

26. 26
NBD (Network) Transport
Lessons From Benchmark Testing

27. NBD (Network) Transport – The Good News
Super easy to set up – 5 minutes on a bad day
No software required on any VMware component
Supports any Datastore type
27

28. NBD (Network) Transport Data Flow – Heat Map
28
LANVMware ESXi
Backup Server
System Load
• ESXi host does all the work
 Pulls data from storage
 Sends data to backup server

29. NBD (Network) Transport – The Not-So-Good
Direct resource impact on ESXi host
Backup data path directly from ESXi host to Backup system
Backup data transfer through VMkernel port
• Our benchmark testing and VMware have verified this
• First backup stream throughput approximately 100MB/sec
• VMware throttles VADP throughput to reserve VMkernel port bandwidth
VMkernel port is bandwidth limited by VMware (e.g. QoS)
We’ve developed techniques for working around this….
29

30. VADP Performance – VMkernel Bandwidth Limits
• Single ESXi host – NBD Performance Per Stream
30
# of streams: 1 2 3 4
100
MB/Sec
100
MB/sec
75
MB/Sec
60
MB/Sec 55
MB/Sec
220 MB/sec
Creates VMkernel
strain on ESXi host
Based on
Cisco/VMware/NetBackup
Benchmark Testing
150
MB/S
Total
100
MB/S
Total
180
MB/S
Total
220
MB/S
Total

31. NBD Best Practices For Optimal Backup Performance
• Number of simultaneous backups?
31
Aggregate Backup Speed
Recommended Settings:
• VMware recommends no more
than 4 backup streams per ESXi
host
• We recommend 1 or 2 streams if
possible
• 10 or more ESXi hosts:
Number of streams = 1
• Less than 10 ESXi hosts:
Number of streams = 2

32. Participate in the discussion!
32
#SYMCHangout
http://bit.ly/1efz3zB

33. SAN Transport
33

34. SAN Transport Data Flow – Heat Map
34
VMware ESXi
Backup Server
System Load
• Skip the ESXi host!

35. SAN Transport – The Good News
Supported with Fibre or iSCSI storage (no NFS support)
Only “off-host” backup for VMware
Near zero impact on ESXi hosts
Backup server does all of the work
More ESXi resources reserved for your VMs
No VMkernel performance limitations
Fastest possible backup performance!
Restore over SAN also possible regardless of backup transport (check with vendor)
35

36. SAN Transport – The Not-So-Good
Storage shared between ESXi host and backup system
Need to understand the risks
Backup system can overwrite VMware partition
36

37. SAN Transport – Best Practices For Protecting Storage
• http://technet.microsoft.com/en-us/library/ff715663.aspx
Use Windows SANpolicy:
Restrict Administrator or Root access on
Windows/Linux backup system
Disable Disk Management snap-in Group Policy
Disable Windows automount on backup system
Configure LUNs as “read only”
37

38. HotAdd Transport
38

39. HotAdd Transport
• Can be fast - bypasses the VMkernel port
• Supports all storage types
• Ideal for remote office environments
The Good News
• Directly impacts the ESXi host with all backup processing
• Proxy VMs need to access the datastore of VMs being
protected
• Configuration issues – add SCSI controllers in VM
• Issues with stuck snapshots, CBT on HotAdd VM, etc.
The Not-So-Good
39

40. How About No Transport?
40

41. Array Based Snapshot – Heat Map
41
VMware ESXi
Backup Server
System Load
• Zero load on any
vSphere component
• No VMkernel
bandwidth limit
• Ultimate performance
1 2

42. Storage Array Based Snapshot – What To Look For
Ability to schedule snapshots just like any other backup
Backup application should automatically handle entire data lifecycle
•Expensive
•Can be required for single file restore
No requirement for additional “backup proxies”
Automatic VM selection based on storage location
•Throttle snapshot creation
•Throttle snapshot deletion
•Critical for limiting I/O “storms”
Control quiesce process – consistent backups
42

43. Storage Array Based Snapshot – What To Look For
Ability to schedule snapshots just like any other backup
Backup application should automatically handle entire data lifecycle
•Expensive
•Can be required for single file restore
No requirement for additional “backup proxies”
Automatic VM selection based on storage location
•Throttle snapshot creation
•Throttle snapshot deletion
•Critical for limiting I/O “storms”
Control quiesce process – consistent backups
43
NetBackup 7.6
NetBackup 7.6
NetBackup 7.6
NetBackup 7.6
NetBackup 7.6

44. NetBackup Replication Director For VMware
44
Protect 300 VMs in 300 Seconds!

45. 45
Recommendations

46. Recommendations - Hardware
• You are paying VMware for CPU & RAM – use these resources for production
Avoid using a VM as backup system
• Two socket, 4 cores per socket plenty of processing power
Modern CPUs are incredibly fast
• 32 GB is minimum
• 64 GB if using deduplication
• More is always better – future proofs this system
RAM
• Performance of internal bus important
• Consider multiple HBA’s for network (teaming) or SAN (DMP)
Pay attention to I/O
46

47. Recommendations - Software
•Spread backups across all Datastores
•Reduces snapshot impact – I/O and space
Limit active backups per Datastore (Resource Limits)
•Avoid spending resources protecting VMs that don’t need it
•Automatically protect VMs that do need it
Use automatic VM backup selection
•Extremely fast
•Be sure solution controls every aspect:
Storage array snapshots add another dimension of protection
•300 VMs in 300 minutes
Backups, restores, snapshot retention, migration to long term storage
47

48. If your using full backups, your doing it the wrong way!
48

49. Advantages of the NetBackup Accelerator for VMware
– Significantly less CPU utilization saves HW and VMware license costs
– Backup is significantly faster – backup more VMs with same/less HW
– Retain all restore functionality – single file, folder, VM, database, application
Backup Processing Impact ESXi Host - CPU Utilization
Standard FULL VADP backup
(NBD transport)
NetBackup Accelerator
For VMware
49

50. Advantages of the NetBackup Accelerator Backup (network)
– Significantly less network traffic on already busy VMkernel port (NIC)
– Less backup hardware required
– Shorter backup window
Backup Processing Impact ESXi Host - Network Utilization
Standard FULL VADP backup
(NBD transport)
NetBackup Accelerator
For VMware
50

51. NetBackup Accelerator for VMware
• Protect extremely large VMs in seconds
• Faster, more reliable backups
51

52. Deduplication Technologies
• Benefits to hard to ignore
• Be careful of solutions that are basically compression
Quickly replacing standard disk as backup target
Trade I/O capacity (expensive) for CPU and RAM (cheap)
• Minimums = 64 GB RAM, 2 Sockets, multiple spindles
BYOD (Bring Your Own Disk) solutions
• NetBackup Appliance – performance tuned
• Hardware configuration correctly designed
• Easy – start protecting your VMs in 10 minutes
Deduplication Appliances
52

53. Thank you!
Copyright © 2013 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in
the U.S. and other countries. Other names may be trademarks of their respective owners.
This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied,
are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.
53
George Winter @vmxgeorge
Abdul Rasheed @AbdulRasheed127