1. ®
Vmware
Automation
Tips, Tricks & Just Good Things to Know
Peter Heigis | Ken Wilson Jan 15, 2008
2. So Here’s The Story
• 1 Company
• 1 Critical Application Set
• ~20,000 users
• COMPLETE REDUNDANCY!
2
3. ®
– Sector: Hospitality – Leader in Hospitality Sales Automation
“If you’ve ever attended a meeting, conference, or special event,
chances are that software from Newmarket International was
behind that experience—from the original specification to the
execution of the individual banquets and functions.”
– Corporate offices in Portsmouth, NH, Singapore, London
– Goal: To provide a completely redundant, highly available, cost
effective, manageable and testable solution that does not
leverage any OS services for replication.
3
4. – Web-based information service that distributes and
manages leads from multiple business channels
– MeetingBroker receives requests for proposals (RFPs) sent
by meeting planners and provides a suite of online tools to
more quickly assign, respond to, and analyze these leads
– Hosted in a well-known data center in Massachusetts
– Utilize Akamai for improved global network performance
4
7. Automate With?
• Automation
– Windows PowerShell & Quest PowerGui
– Virtual Infrastructure Management
Assistant (VIMA) Appliance
Automation
– Perl Toolkit
– Custom Bash & Perl Scripting
– Cluster Services
– SAN Based Tools
– Hardware Based Features
Through Innovation
Call Home, RSA etc…
– Augmenting Applications
Secondary Vmware Applications 7
8. Automate How Long?
• Phased Approach
– Capacity Planner Assessment
35 Days = TCO, ROI & Consolidation Scenario
– Design Blueprint
10-15 Days - Detailed Installation and
Configuration
– Installation
5 Days- 5 IBM 3850, 2 DS-4X00
– Testing & Mitigation
5 Days -Hardware Failover, DR Failover
– Documentation
2-3 Days = SOP(s), VDOC
8
9. Automation Methodology
• KISS
• Leverage Freeware
• Constant Re-evaluation
• Quick Learning Curve
• Scalable after the project
• Testable
9
10. Issues to Address
• How do you automate DR without SRM?
• How do you know the configuration of PCI
devices?
• How do you deal with VMDK files located on
different LUN(s)?
• How do you handle tiered network architecture?
• How do you install ESX with attached FC?
• How does this change your backup strategy?
10
12. What Was Used?
• BIOS & Firmware Updates
– IBM Auto Update
• Installation & Configuration using
– ESXi
– Kickstart
– RSA
• VM Deployment and Migration
– Windows PE
– Vizioncore
• High Availability & Failover
– HA & DRS (With Rulesets)
– SAN Based Replication
– Scripts
• Documentation
12
19. Install or Recover
esx noprobe ks=hd:scd0:/kickstartfile.cfg
Or if you don’t have Remote Control
esx noprobe ks=hd:scd0:/kickstartfile.cfg
19
20. Great… Where Do I Get IT
• IZARC = Free (www.izarc.org)
•HP Drive Key Boot Utility = Free (www2.hp.com)
•ESXi = Free (www.vmware.com)
•Magic ISO = $29.95 (www.magiciso.com)
20
22. OK DOC… What Do I Got
• Performance
– For optimum I/O performance, the file
systems of the VMDK, the datastore and the
storage must be in proper alignment
• What’s The Deal
– Windows and Linux systems reserve space at
the beginning of a disk in order to store their
boot information (MBR partition)
– Windows assumes 63 sectors per Track
Reserves Track 0 – Sectors 0-62
– Storage Arrays assume 64 sectors per Track
• The Result
– Partition starting offset is on Track 0 Sector
63 and continues onto Track 1
– Misaligned I/O due to Track crossing
22
23. Impact of Misalignment
• Performance impact of misalignment
– Varies based on workload
4K I/O 1/8 of IOs misaligned
8k I/O 1/4 of I/Os misaligned
16k I/O 1/2 of I/Os misaligned
32k I/O every I/O misaligned
– Performance hit dependent on I/O size
2-3% up to 60%
• Ask customers to correct their VM templates
– Make future VMs perform optimally
23
24. How Does This Happen
• VMFS partitions created via vmkfstools will always be
misaligned
– Use fdisk to check
# fdisk –lu /dev/sd*
• VMFS partitions created via VI Client will always be
aligned
24
25. VMDK Alignment Issue
• By default operating systems reserve space at the
beginning of a disk in order to store their boot
information = “the partition offset”.
• To build a VMDK:
1. Start with a VMFS Volume/LUN.
2. Create a VMDK file
3. Create a Partition and a Filesystem (within the VMDK).
Windows.vmdk
NTFS
VMFS Volume/LUN 25
26. Default Partition Offset
• Both NTFS and VMFS ultimately store blocks.
• The default partition starts at an offset of 63
sectors—32256.
• Read 1 Block within the Virtual Machine
• Requires 2 IO operations
This can cause performance issues!
NTFS
VMFS
26
27. VMDK Misalignment Issue is
Vendor Neutral
• The alignment issue impacts all storage vendors
– Most are unaware of this issue
• Misaligned VMDKs is consistently a top 10 customer
support issue, reports VMware
27
30. Pre-emptive Alignment Mitigation
c:diskpart
NTFS
DISKPART> list disk
Disk ### Status Size Free Dyn Gpt
-------- ---------- ------- ------- --- ---
Disk 0 Online 4095 MB 8033 KB
Disk 1 Online 4103 MB 4103 MB
DISKPART> select disk 1
Disk 1 is now the selected disk.
VMFS
DISKPART> create partition primary align=32 (64 for sql & exchange)
DiskPart succeeded in creating the specified partition.
30
31. Reactive Alignment Mitigation
• You Had Two Options Prior to VM Creation:
– Create VMDK, connect to existing VM for partition creation
– Create VMDK, assign to new VM, boot with Windows
Preinstall Environment CD for partition creation
• In the Old Days…You Had One Option for Alignment
– Requires Data migration
• Now You Can Do It…..Gui
31
33. Automate DR?
• Success Criteria
– No Dependency on Windows based Services
– Completely Redundant
– Testable
• Solution
– (2) IBM DS 4700
– Storage Based Replication (Async)
Does not protect against corruption
• Issues
– VMFS Presentation
– VM Registration
UUID… “To Some its not a Problem”
33
34. VMware DR w/SRM
Primary Site Secondary Site
Virtual Center&
Virtual Center&
VMware View Site Recovery
Site Recovery VMware View
Manager
Manager
Array Replication
Datastores Datastores
Virtual
Desktop
Virtual
Desktop
34
35. Newmarket Complete Redundancy
Primary Site Re-Register VM(s)
Registered VM(s)
dc.vmdk
dc.vmdk Virtual Center
Iis.vmdk
Iis.vmdk
Mount Replicated LUN(s)
Array Replication
Datastores Datastores
35
38. How We Got Around IT?
•Treated the Replication like a Snapshot
•Simple Script
•Searches all VMs for hardcoded UUID
•Creates an Array of UUID to Common Volume Names
•Comments out old entry: Replace UUID with Common Volume
Name
•Automated through cron
38
39. How Backups are Affected?
•No longer necessary to backup the System Disk every week
•Only need to backup changed files (Incremental Forever)
•Leverage replicated snapshots for quick file level restore
39
40. What We Used
As-Built Configuration Change
Documentation Control
40
