Deep dive in App Volumes troubleshooting

1. © 2015 VMware Inc. All rights reserved.
VMware App Volumes Troubleshooting
Denis Gundarev
Sr. MTS/Application Architect, App Volumes R&D

2. About me
C:>whoami /all
USER INFORMATION
----------------
User Name Twitter E-Mail
================ ============ ==================
VMWAREdgundarev @fdwl DGundarev@vmware.com
GROUP INFORMATION
-----------------
Group Name Type SID
======================================== ================ =================
BUILTINGeeks Mandatory group S-1-5-32-540
Mandatory LabelCrazy Russian Label S-1-16-8192
VMWAREApp Volumes R&D Mandatory group S-1-5-32-544

3. App Volumes Dynamic Delivery Benefits
Dynamic delivery of managed application containers in seconds.
Real time native application and data delivery.
Agility
– Logically manage application sets
based on business needs.
– Deliver or upgrade application
sets across all VMs in seconds.
Simplicity
– Integrate into existing
infrastructure in minutes.
– Provision applications as easily as
installing them.
Flexibility
– Persistent user experience with
non-persistent economics.
– Works with VMware Horizon 6
with View and Citrix XenApp 6.5.
Efficiency
– Optimize use of storage, SAN
IOPS, and network.

4. App Volumes
• Feb 3, 2015 - VMware App Volumes v2.5
– Horizon View integration
– Multiple templates for writable volumes
• Mar 12, 2015 - VMware App Volumes v2.6
– AppStack Grouping
– Storage Groups
• Apr 28, 2015 - VMware App Volumes 2.7
– One-way AD trusts
– Dynamic VHD permissions
– XenDesktop 7.x support
• Jun 15, 2015 - VMware App Volumes 2.9
– Support for application deployment to physical machines
– Multi vCenter Configurations

5. App Volumes components
App Volumes Manager
App Volumes Agent
AppStack volume(s)
• Console for assignments and configuration
• Broker for App Volume Agent for the assignment of
applications and writable volumes
• File system and registry abstraction layer running on
the target system
• Virtualizes file system writes as appropriate (when
used with optional writable volume)
• Read-only volume containing applications
• Can map more than one AppStack per user, target
• Deploy apps to VDI or RDSH
Writable volume • Per user read-write volume used to persist
changes written in the session
• One writable volume per user

6. vCenter
App Volumes Architecture
vSphere ESXi
SOAP
App Volumes Manager
NGINX
HTTP/HTTPS
(80/443)
HTTPS (443)
hostd
SQL Database
Virtual Machine
Minifilter driver
Ruby
svmanager
TCP 1433
HTTPS (443)
Domain Controller
LDAP (389) /
LDAPS (636)
Connection Server
Broker
ServiceHTTP/HTTPS
(80/443)
HTTP
Manager (Web
Interface)
API
Scheduler Workers
Service
Base OS AppStack AppStack

7. App Volumes Operation Mode
• VMDK Direct Attached Operation Mode
– preferred operations mode for App Volumes
– AppStacks and Writable Volumes are stored
within a hypervisor datastores
– VMDK file format
– attached to the virtual machine using standard
hypervisor functionality.
• VHD In-Guest Operation Mode
– AppStacks and Writable Volumes are stored
on the (CIFS) file share
– VHD file format
– attached to the target computer using OS
functionality
– Works with physical and other hypervisors

8. VHD In-Guest
• VHD AppStacks mounted inside the target
computer, similar to disk manager command
• Target computer should have access to the
file share
• VHD file permissions can be dynamically
controlled by the manager
• Works with physical and third-party
hypervisors

9. Hypervisor Connection Type
• Connection to VMware vSphere vCenter
– preferred connection type for most environments
– VMDK Direct Attached Operation Mode
• ESX (Single Host) Connection
– VMDK Direct Attached Operation Mode
– works only for a single hypervisor host
– use for small deployments and PoC
• VHD In-Guest
– disables hypervisor connection
– enables use of VHD In-Guest Operation Mode
CONFIDENTIAL 9

10. App Volumes - Scalability
• Observe vSphere 5.5 Maximums
– 2048 virtual disks per host
– 2048 powered-on virtual machines per VMFS volume
– 60 SCSI devices per virtual machine
• Recommended Practice:
– 1 AppStack per 2,000 virtual machines
– Up to 20 AppStack volumes per VM
– 10,000 Agents per App Volume Manager
• Core applications
– Part of the base image or single AppStack
– AppStack datastore for every 2,000 users

11. AppStack
• App Volumes is not ThinApp Alternative
– “ghost” for applications
– Works with ThinApp or App-V
• The contents of the system drive (C:) located under SVROOT in the VHD.
– Only file system entries that had contents changed / updated or security information changed / updated.
– The files are ordered in the same directory tree as in the source machine
– All the file system entries have the original security information / attributes.
• The registry changes are located in snapvol.dat and have a valid Windows registry hive
formatting.
– The changes / updates to the SOFTWARE hive are under MACHINESOFTWARE key in the hive.
– The changes / updates to the SYSTEM hive are under MACHINESYSTEM key in the hive.
– The changes / updates to the DEFAULT hive are not captured as part of the VHD.

12. Configuration rules

13. Workflows
twitrcovers.com
When Stuff Happens

14. App Volumes Provisioning Volumes Workflow
Target System
App Volumes Manager
vCenter
1. Configuration of AppStacks
2. Mount Command sent to vCenter
- Target system
- VMDK information
- DataStore Information
3. Target system receive AppStacks
1
2
3

15. Login
App Volumes Agent Events
Logout
Startup Shutdown
43
21

16. App Volumes Agent Startup
App Volumes Manager
Startup
1. App Volumes Agent checks into App Volumes Manager
2. App Volumes Manager provides license and machine
assignments
2
App Volumes Agent
1

17. App Volumes Agent Shutdown
App Volumes Manager
Shutdown
1. App Volumes Agents checks into App Volumes Manager
2. User based assignments are detached
2
App Volumes Agent
1

18. App Volumes Agent Login
App Volumes Manager
Login
1. App Volumes Agent checks into App Volumes Manager
2. App Volumes Agent checks for pending attachments/un-attachments
3. App Volumes Agent checks for machine based attachments
– if present then no user based attachments are honored
4. App Volumes Manager checks database for logged in user assignments
5. Attach assigned volumes - Writable then AppStack(s)
2
App Volumes Agent
1
Database
3
4
5

19. App Volumes Agent Logout
App Volumes Manager
Logout
1. App Volumes Agents checks into App Volumes Manager
2. User based assignments are detached
2
App Volumes Agent
1

20. App Volumes Provisioning
• Start with Clean VM image
• Create AppStack in App Volumes Manager
• Choose Provisioning VM
• VMDK gets attached to Provisioning VM
• Install application
• Complete AppStack creation
• VM restarts, VMDK is detached
• Ready to assign AppStack

21. Scripts
Script name Execution condition
prestartup.bat Called when a volume is dynamically attached or on during system startup but before virtualization
is activated
startup.bat Called when a volume is dynamically attached or on during system startup
startup_postsvc.bat Called as and called after services have been started on the volume (not called if there are no
services on volume)
logon.bat Called at logon and before Windows Explorer starts
logon_postsvc.bat Called after services have been started (not called if there are no services on volume)
shellstart.bat Called when a volume is dynamically attached or when Windows Explorer starts.
shellstop.bat Called when the user is logging off before Windows Explorer is terminated.
logoff.bat Called at logoff and Windows Explorer has terminated
shutdown_presvc.bat Called when the computer is being shutdown before services have been stopped.
shutdown.bat Called when the computer is being shutdown after services have been stopped.
allvolattached.bat Called after all volumes have been processed (so if user has 3 AppStack, this will be called after all
3 have loaded)
post_prov.bat Called at the end of provisioning to do any one-time steps that should be performed at the end of
provisioning. Invoked at the point of clicking the provisioning complete pop-up while the volume is
still virtualized.
prov_p2.bat Invoked at phase 2 of the provisioning process. After the machine has rebooted but before App
Volumes Manager has been notified that provisioning has completed. This is the last chance to
perform any actions on the provisioned volume with virtualization disabled.

22. Troubleshooting

23. App Volumes – Agent Log files
• App Volumes Agent
– C:Windowssvservice.log
– SvService responsible for communication with
App Volumes Manager, preparing volume,
running post-attach scripts, refreshing
variables, registering fonts
• App Volumes Agent
– C:WindowsSystem32LogFilesWMI
AppVolumesAgent.etl
– Minifilter drivers for NTFS
– Policy driven; the files, directories, registry keys
and processes that are virtualized are governed
by the policy file snapvol.cfg on each volume.
• App Volumes Agent svcapture.log
– C:Windowssvcapture.log
– Perform the provisioning/editing functions,
generate policy files, metadata and report it to
Agent/Manager.

24. Reading SVDriver log
• Converting an ETL file to xml format
– Flush buffers to disk
• logman update AppVolumesAgent -fd –ets
– Convert log to xml
• tracerpt.exe AppVolumesAgent.etl -of xml -gmt -tp “C:Program Files (x86)CloudVolumesAgenttmf” -o AppVolumesAgent.xml
– Open AppVolumesAgent.xml using Excel
• Event Viewer:
– Copy TMF files from C:Program Files (x86)CloudVolumesAgenttmf to
C:windowsSystem32WinevtTraceFormat
– Open "C:windowsSystem32LogFilesWMIAppVolumesAgent.etl“ using windows Event Viewer

26. App Volumes Manager – Production Log
• The Production.log file can help
identify issues
• You can view the Production.log at
http://<App Volumesmgr>:8080/log
• The default log level is set to ‘info’.
• Logfile is also located at:
C:Program Files
(x86)CloudVolumesManagerLog
production.log

27. ESXi Logs
• ESXi Host Management Service Log (hostd) /var/log/hostd.log
– Watch for disk locking, VM state transition, hot-add operations
• Virtual Machine Log /vmfs/volumes/<datastore>/<vm_name>/vmware.log

28. • Debug provides SQL logging
App Volumes Manager – Increasing Logging Level
• We can increase the logging level to debug, by editing:
C:Program Files(x86)CloudVolumesManagerconfiglog4r.yml

29. Example 1: Provision a new App Stack
CONFIDENTIAL
30
Template VMDK copied
to new App Stack VMDK
UUID of clean machine to
mount App Stack for
provisioning
App Volumes module
initializes capture,
RvSphere module mounts
volume
Operation: New App Stack ‘Office2010_RTM’ created
Where to Look: App Volumes Manager Log
C:Program Files (x86)CloudVolumesManagerlogproduction.log

30. Example 1: Provision a new App Stack (continued)
CONFIDENTIAL
31
VM State transitions during
operation:
VM_STATE_ON 
VM_STATE_RECONFIGURED
VM_STATE_RECONFIGURING

VM_STATE_ON.
Hot Add completed and App
Stack mounted on VM
Operation: New App Stack ‘Office2010_RTM’ created
Where to Look: ESXi hostd.Log: /var.log.hostd.log
Relevant Log entries: Hot-Add, VM transition, disk locking.

31. CONFIDENTIAL
32
Example 2: SQL Database Unreachable
Behaviour: Manager: UI is unavailable – CVManager.exe service stays up, attempting
reconnection. ODBC error in Manager.log.
Agent: Where user remains logged in - App Stacks / Writables work as normal
For new logins - App Stacks / Writables work as normal
Issue: SQL Server Database is unreachable by App Volumes Manager(s)
Extract from
App Volumes
Manager.log
Where to Look: App Volumes Manager Log
C:Program Files (x86)CloudVolumesManagerlogproduction.log
Relevant Module entries: ‘ODBC’

32. CONFIDENTIAL
33
Example 3: AD Unreachable
Issue: Active Directory Domain Services unreachable (including DNS)
Extract from
App Volumes
Manager.log
Behaviour: Manager: UI is functional, but AD queries fail – error in Manager.log
Agent: Where user remains logged in - App Stacks / Writables work as normal
For new logins - App Stacks / Writables unavailable
Where to Look: App Volumes Manager Log
C:Program Files (x86)CloudVolumesManagerlogproduction.log
Relevant Module entries: ‘RADIR’

33. CONFIDENTIAL
34
Example 4: vCenter Unreachable
Issue: vCenter unreachable
(Note: Mount on ESXi host option was set during install)
Extract from
App Volumes
Manager.log
Behaviour: Manager: UI is partially functional, but error on querying
datastores. Unable to create new App Stacks/Writables
Agent: App Stacks / Writables fully functional
Where to Look: App Volumes Manager Log
C:Program Files (x86)CloudVolumesManagerlogproduction.log
Relevant Module entries: ‘RvSphere’

34. App Volumes Manager – Ruby Console
• The Ruby on Rails Console can be
used to troubleshoot, execute queries
and run scripts
– Common use to examine internal
objects not visible in the UI
• Open command prompt as
Administrator on App Volumes Manager
machine
• Navigate to:
C:Program Files
(x86)CloudVolumesManager
Enter the command:
rubybinruby.exe scriptrails console
35
Any ruby command
should be used
with CAUTION ! ! !

35. App Volumes Manager – Ruby Console
• List registered App Volume Machines:
– ap Machine.all
• Show configuration
– ap Svconfiguration.current
• Delete pending activities
– Delayed::Job.destroy_all
• Search for machines
– ap Machine.where('name LIKE ?', "PVS%")
• Delete all disabled computers
– ap Computer.where("disabled_at IS NOT NULL").delete_all
36

36. App Volumes Manager – Ruby Console
• List admin groups
– ap GroupPermission.all
• Delete all administrators
– GroupPermission.delete_all
• Add administrator group
– Permission.named(:admin).add_group_name("CN=Domain Admins,OU=Groups,
DC=test,DC=com")
37

37. App Volumes Manager PowerShell
• Initiate session
– $url="http://appvol01.corp.itbubble.ru"
– $person = @{'user[account_name]'=‘admin'; 'user[password]'='P@ssw0rd'}
– Invoke-RestMethod -Uri $url/login -Method POST -SessionVariable session -Body
$person
• Get version
– Invoke-RestMethod -Uri $url/login -Method POST -SessionVariable session -Body
$person
• Get online users
– (Invoke-RestMethod -Uri $url/cv_api/online_entities -WebSession $session -Method
GET).online.records
• Get configuration
– (Invoke-RestMethod -Uri $url/cv_api/summary -WebSession $session -Method
GET).current_configuration
38

38. AppVolumes PowerShell Module Preview
• Open-AppVolSession
• Test-AppVolSession
• Close-AppVolSession
• Get-AppVolAppStack
• Get-AppVolAppStackFile
• Get-AppVolAssignment
• Get-AppVolComputer
• Get-AppVolGroup
• Get-AppVolOrgUnit
• Get-AppVolUser
• Get-AppVolVersion

39. AppVolumes PowerShell Module Preview
• Open-AppVolSession -Uri http://manager.domain.com -Username admin -Password
password1
• Or you can omit the parameter names:
• Open-AppVolSession http://manager.domain.com admin password1
•

40. AppVolumes PowerShell Module Preview
• Examples:
• Get-AppVolAppStack [-All] - Returns all available appstacks
• Get-AppVolAppStack -AppStackIds 88,19 – return appstacks with IDs 88 and 19
• 88,19|Get-AppVolAppStack – return appstacks with IDs 88 and 19 thru the pipe
• Get-AppVolAppStack -Name office – returns all appstacks where the name contains “office”
• Get-AppVolAppStack -Name office -Not - returns all appstacks where the name NOT contains “office”
• Get-AppVolAppStack -Name office -Exact – all apstacks where the name is exactly “office”
• Get-AppVolAppStack -Path "cloudvolumes" – appstacks that has “cloudvolumes” in the datastore path
• Get-AppVolAppStack -DataStore iSCSI -Exact – appstacks located on datastore iSCSI
• Get-AppVolAppStack -FileName office – appstacks where vmdk name contains word office
• Get-AppVolAppStack -CreatedAt "4/28/2015" –ge – appstacks created after or on 4/28/2015
• Get-AppVolAppStack -CreatedAt "4/28/2015" -gt appstacks created after but not on 4/28/2015
• Get-AppVolAppStack -MountedAt $((get-date).AddDays(-30)) -ge -Not - appstacks not mounted in past 30 days
• Get-AppVolAppStack -TemplateVersion "2.5.1" – appstacks with template version 2.5.1
• Get-AppVolAppStack -AssignmentsTotal 2 -ge - appstacks that have 2 or more assignments
•

41. AppVolumes PowerShell Module Preview
• Get-AppVolAppStackFile -DataStore datastore1|Get-AppVolAppStack|Format-
Table returns a table with all appstacks that have files on datastore1
• Get-AppVolAppStackFile -Reachable -Not -returns all unreachable files
• Get-AppVolAppStack -Name office |Get-AppVolAssignment - get all assignments for
appstacks that has “office” in the name
• Get-AppVolAssignment -EntityDn "cn=users,dc=domain,dc=com" – assignments for users
in specific OU
• Get-AppVolAssignment -EntitySamAccountName denis -Exact – assignments for user
“denis”
•

42. App Volumes Agent - Registry Entries
HKLMSoftwareCloudVolumesAgent
HKLMSystemCurrentControlSetservicessvserviceparameters
• vCenter / App Volumes
Manager Hostname and Port
• Additional App Volumes Managers
– registry entries need to be added
manually after standard install
– Can be set during Msiexec install
MANAGER_ADDR=test.company.c
om MANAGER_PORT=80

43. App Volumes & MSOffice

44. Microsoft Office Support Basics
• We don’t officially support scenarios that are not supported by Microsoft
– Office 2007
– Mix of 64-bit and 32-bit apps
• We only support:
– Office Professional Plus
– Office Standard
– Office 365 ProPlus (installed from Office Professional Plus media)
• Only media downloaded from Microsoft Volume Licensing Service Center (VLSC) can be used
– No MSDN
– No Retail/OEM
– Verify if your installation media is correct:
• You must have a folder named “Admin” on your ISO
• Name of the folder with .WW extension (ProPlus.WW, Visio.WW, etc.) shouldn’t have a lowercase “r” before the
dot - ProPlusr.WW, Visior.WW

45. Microsoft Office Activation
• KMS should be used for most deployments
• MAK activation can be used for persistent deployments starting with 2.7
• Retail, OEM, Office 365 activation not supported
• Active Directory-Based Activation (Office 2013 on Windows 8) may work, but not tested
• KMS DNS discovery preferred
• Rearm Office after installation
• If activation fails, verify that KMS activation works for a machine with no App Volumes agent

46. Microsoft Office Recommendations
• Recommend Office in the base image
• One AppStack for app Office Applications
• Never launch Office Apps during provisioning
• Rearm Office after installation
• Add-ins/plug-ins must be on the same AppStack with Office
• Recommend 32-bit editions
• 32-bit and 64-bit editions should never meet, even from different AppStacks

47. Multiple Office Versions Within AppStack
Office 2010 Office 2013 Project 2010 Project 2013 Visio 2010 Visio 2013
Office 2010 N/A
Install Office
2010 first
Supported
Install Office
2010 first
Supported
Install Office
2010 first
Office 2013
Install Office
2010 first
N/A
Install Project
2010 first
Supported
Install Visio 2010
first
Supported
Project 2010 Supported
Install Project
2010 first
N/A
Install Project
2010 first
Supported
Install Project
2010 first
Project 2013
Install Office
2010 first
Supported
Install Project
2010 first
N/A
Install Visio 2010
first
Supported
Visio 2010 Supported
Install Visio 2010
first
Supported
Install Visio 2010
first
N/A
Install Visio 2010
first
Visio 2013
Install Office
2010 first
Supported
Install Project
2010 first
Supported
Install Visio 2010
first
N/A

48. Native Office Coexistence
AppStack
Office 2010 Office 2013
Project
2010
Project
2013 Visio 2010 Visio 2013
Nativelyinstalled
Offic
e
2010 Not Supported
Supported
except Outlook Supported Supported Supported Supported
Offic
e
2013 Not Supported Not Supported
Not
Supported Supported
Not
Supported Supported

49. Thank You!
Questions?