1. Citrix Internals: Tracing, Debugging &
Troubleshooting
(The Shaman’s guide to troubleshooting)
Denis Gundarev
Senior Consultant
Entisys Solutions

2. About me
C:>whoami /all
USER INFORMATION
----------------
User Name Twitter E-Mail
============== ============ ==================
ENTISYSdenisg @fdwl DenisG@entisys.com
GROUP INFORMATION
-----------------
Group Name Type SID
======================================== ================ =================
BUILTINGeeks Mandatory group S-1-5-32-540
Mandatory LabelCrazy Russian Label S-1-16-8192
COMMUNITYBay Area Citrix User Group Well-known group S-1-5-32-544
COMMUNITYCitrix Technology Professional Well-known group S-1-5-32-545

3. What’s the topic?
• Not a basic “Restart the computer” stuff
• Do not restart the computer if you going to troubleshoot the issue
• Not a CDF tracing tutorial
• Search for TechEdge presentations on citrix.com
• Not a crash dump analysis workshop
• Check Dmitry Vostokov’s http://www.dumpanalysis.org/

5. Shaman’s Guide to Troubleshooting
Phase 1 Phase 2 Phase 3
Identify
the Issue
Profit

7. StoreFront Troubleshooting

8. support.citrix.com
• 250+ articles about Web Interface
• 2000+ articles about XenApp
• 240 articles about XenDesktop 7
• 8 articles about StoreFront 2.0

9. Generic Errors in StoreFront

11. Step 1: Check event logs

12. Storefront Internals
• StoreFront (as well as most of the XenDesktop components) is based
on Windows Communication Foundation (WCF)
• WCF services communicate with each other using endpoints
• Endpoint configuration and listening address defined in the .config file

13. Step 2: Search files for text
PS C:>Get-ChildItem -Recurse *.config|Select-String "net.pipe://localhost/Citrix/Authentication/Win32"
Select-Object -Property Path, LineNumber
|
|Out-GridView -OutputMode Multiple |%{notepad $_.Path}

14. Some service is causing this issue. Which one?

15. Step 3: Search for service by executable path
PS C:>Get-WmiObject -Class Win32_Service -Filter "pathname like '%defaultdomainservices%'"
Select-Object -Property Displayname, PathName, Started
|
|Out-GridView

16. Generic Errors in StoreFront

17. Step 1: Check event logs

18. Step 2: Search files for text
PS C:>Get-ChildItem -Recurse *.config|Select-String "loadbalancer.corp.itbubble.ru"
Select-Object -Property Path, LineNumber
|
|Out-GridView -OutputMode Multiple |%{notepad $_.Path}

19. Step 2: Search files for text
PS C:>Get-ChildItem -Recurse *.config|Select-String "loadbalancer.corp.itbubble.ru"
Select-Object -Property Path, LineNumber
|
|Out-GridView -OutputMode Multiple |%{notepad $_.Path}

20. Fiddler

21. Generic Errors in StoreFront

22. Fiddler

23. Fiddler
• Set-Cookie:
NSC_ttmx=ffffffffc3a01fa645525d5f4f58455e445a4a423660;expires=Sun, 14-
Jul-2013 13:46:47 GMT;path=/;secure;httponly
• Set-Cookie:
NSC_ttmx=ffffffffc3a01fa245525d5f4f58455e445a4a423660;expires=Sun, 14-
Jul-2013 13:47:15 GMT;path=/;secure;httponly
• Set-Cookie:
NSC_ttmx=ffffffffc3a01fa645525d5f4f58455e445a4a423660;expires=Sun, 14-
Jul-2013 13:47:15 GMT;path=/;secure;httponly
• Set-Cookie:
NSC_ttmx=ffffffffc3a01fa245525d5f4f58455e445a4a423660;expires=Sun, 14-
Jul-2013 13:47:15 GMT;path=/;secure;httponly

25. StoreFront tracing
• http://support.citrix.com/proddocs/topic/dws-storefront-10/dws-
troubleshoot.html
• To enable tracing
• Using an account with local administrator permissions on the Receiver
Storefront server, start Windows PowerShell and, at a command prompt, type
the following commands.
• > Add-PSSnapin Citrix.DeliveryServices.Framework.Commands
• > Set-DSTraceLevel -All -TraceLevel Verbose
• To disable tracing, type the following commands.
• > Add-PSSnapin Citrix.DeliveryServices.Framework.Commands
• > Set-DSTraceLevel -All -TraceLevel Off

26. StoreFront tracing

27. Service Configuration Editor
• Available with Windows
SDK
• Allows administrators to
modify configuration
settings for WCF services
using a graphical user
interface

28. Search Files for Text
PS C:>Get-ChildItem -Recurse *.config -ErrorAction SilentlyContinue |
|Select-Object -Property Path,LineNumber |Select-String "Program FilesCitrixReceiver StoreFrontAdminTrace"
Out-GridView -OutputMode Multiple |%{c:SvcConfigEditor.exe $_.Path}

29. Set up XML tracing
• Diagnostics -> Listeners -> SetDSTraceLevelCmdletListener

30. Set up XML tracing
• InitData -> c:temp<ServiceName>.svclog
• TypeOutputOptions ->
-> Check all fields
• TypeName ->
System.Diagnostics.XmlWriterTraceListener

31. Service Trace Viewer Tool (SvcTraceViewer.exe)

32. Service Trace Viewer Tool (SvcTraceViewer.exe)

33. Receiver for Web

34. Receiver for Web

35. 35

36. 36

37. 37

38. 38

39. 39

40. Reverse Engineering

41. 41
ILSpy
• Assembly browsing
• IL Disassembly
• Decompilation to C#
• Decompilation to VB
• Find usage of field/method
• Available at http://ILSpy.net
• Free

42. 42
• Unhandled exception error when browsing for an application in
Delivery Services Console or AppCenter

43. 43
• System.IO.DirectoryNotFoundException: The system cannot find the path specified. (Exception from HRESULT:
0x80070003)
• at IWshRuntimeLibrary.IWshShortcut.Save()
• at Citrix.CMI.PSE.Utils.FileBrowsing.CreateServerLinks(FileDialog dialog, ServerSet serverSet, ModifyShareNameCallback
modifyShareNameCallback)
• at Citrix.CMI.PSE.Cmo.App.PropertyPages.InstalledICALocationPage.u_browseCommandLineButton_Click(Object sender,
EventArgs e)
• at System.Windows.Forms.Control.OnClick(EventArgs e)
• at System.Windows.Forms.Button.OnClick(EventArgs e)
• at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
• at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
• at System.Windows.Forms.Control.WndProc(Message&#38; m)
• at System.Windows.Forms.ButtonBase.WndProc(Message& m)
• at System.Windows.Forms.Button.WndProc(Message8; m)
• at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message&; m)
• at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
• at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

44. 44
• Cause:
• Customer have Folder redirection implemented via GPO and
"Links" special folder is redirected.
• Citrix Management console doesn't handle folder redirection and
assume that %USERPROFILE%Links folder exists.
•
private static readonly string FavoriteLink =
Path.Combine(Path.Combine(Environment.GetEnvironmentVa
riable("USERPROFILE"), "Links"),
Services.ResString.GetGlobal("$Profile_Links_Shortcut_Nam
e") + ".lnk");

45. 45

46. 46
Search Files for Text
PS C:>Get-ChildItem -Recurse *.* -ErrorAction SilentlyContinue |Select-String "XenDesktop is not installed"
Select-Object -Property Path
|
C:WindowsassemblyNativeImages_v2.0.50727_32Citrix.GroupPolicy.#
a4a323e135cabd1324a46585b70ee461Citrix.GroupPolicy.Filters.ni.dll

47. 47

48. 48

49. 49

50. 50

52. Is it Legal?
In any event, you may not modify, translate, reverse engineer, decompile,
disassemble, create derivative works based on, or copy the SOFTWARE,
except as specifically licensed herein or to the extent such foregoing restriction
is expressly prohibited by applicable law.“ Citrix XenDesktop 7 EULA
Are there things I’m not allowed to do with the software? Yes. Because the
software is licensed, not sold, Microsoft reserves all rights ..... not expressly
granted in this agreement. In particular, this license does not give you any right
to, and you may not: ..... reverse engineer, decompile, or disassemble the
software, except if the laws where you live permit this even when our
agreement does not. In that case, you may do only what your law allows. .....
Microsoft Windows 8 EULA
In any event, you may not modify, translate, reverse engineer, decompile,
disassemble, create derivative works based on, or copy the SOFTWARE,
except as specifically licensed herein or to the extent such foregoing restriction
is expressly prohibited by applicable law.
Are there things I’m not allowed to do with the software? Yes. Because the
software is licensed, not sold, Microsoft reserves all rights ..... not expressly
granted in this agreement. In particular, this license does not give you any right
to, and you may not: ..... reverse engineer, decompile, or disassemble the
software, except if the laws where you live permit this even when our
agreement does not. In that case, you may do only what your law allows. .....
In any event, you may not modify, translate, reverse engineer, decompile,
disassemble, create derivative works based on, or copy the SOFTWARE,
except as specifically licensed herein or to the extent such foregoing restriction
is expressly prohibited by applicable law.
Are there things I’m not allowed to do with the software? Yes. Because the
software is licensed, not sold, Microsoft reserves all rights ..... not expressly
granted in this agreement. In particular, this license does not give you any right
to, and you may not: ..... reverse engineer, decompile, or disassemble the
software, except if the laws where you live permit this even when our
agreement does not. In that case, you may do only what your law allows. .....

53. Is it Legal?
Sec. 103(f) of the DMCA (17 U.S.C. § 1201 (f)):
• (f) Reverse Engineering.—
(1) Notwithstanding the provisions of subsection (a)(1)(A), a person who has lawfully obtained the right to use a copy of a computer program may circumvent a technological measure that effectively controls
access to a particular portion of that program for the sole purpose of identifying and analyzing those elements of the program that are necessary to achieve interoperability of an independently created computer
program with other programs, and that have not previously been readily available to the person engaging in the circumvention, to the extent any such acts of identification and analysis do not constitute
infringement under this title.
(2) Notwithstanding the provisions of subsections (a)(2) and (b), a person may develop and employ technological means to circumvent a technological measure, or to circumvent protection afforded by a
technological measure, in order to enable the identification and analysis under paragraph (1), or for the purpose of enabling interoperability of an independently created computer program with other programs,
if such means are necessary to achieve such interoperability, to the extent that doing so does not constitute infringement under this title.
(3) The information acquired through the acts permitted under paragraph (1), and the means permitted under paragraph (2), may be made available to others if the person referred to in paragraph (1) or (2), as
the case may be, provides such information or means solely for the purpose of enabling interoperability of an independently created computer program with other programs, and to the extent that doing so does
not constitute infringement under this title or violate applicable law other than this section.
(4) For purposes of this subsection, the term 「interoperability」 means the ability of computer programs to exchange information, and of such programs mutually to use the information which has been
exchanged.
Article 6 of the 1991 EU Computer Programs Directive
• (15) The unauthorised reproduction, translation, adaptation or transformation of the form of the code in which a copy of a computer program has been made available constitutes an infringement of the
exclusive rights of the author. Nevertheless, circumstances may exist when such a reproduction of the code and translation of its form are indispensable to obtain the necessary infor-mation to achieve the
interoperability of an indepen-dently created program with other programs. It has therefore to be considered that, in these limited circum-stances only, performance of the acts of reproduction and translation by
or on behalf of a person having a right to use a copy of the program is legitimate and compatible with fair practice and must therefore be deemed not to require the authorisation of the right-holder. An objective
of this exception is to make it possible to connect all components of a computer system, including those of different manufacturers, so that they can work together. Such an exception to the author's exclusive
rights may not be used in a way which prejudices the legitimate interests of the rightholder or which conflicts with a normal exploitation of the program.
“ …for the sole purpose of identifying and analyzing
those elements of the program that are necessary to
achieve interoperability of an independently created
computer program with other programs…
…to obtain the necessary infor-mation to achieve
the interoperability of an indepen-dently created
program with other programs….

54. Conclusion
• Use PowerShell for searching
• Analyze config files
• Use Fiddler to sniff StoreFront traffic
• Use ILSpy

55. Q&A
• @fdwl
• denisg@entisys.com
• http://meetup.com/BayCUG
• http://blog.itbubble.ru