Debugging tools used for production debugging of web applications

1. © Copyright SELA Software & Education Labs Ltd. | 14-18 Baruch Hirsch St Bnei Brak, 51202 Israel | www.selagroup.com
SELA DEVELOPER PRACTICE
December 11-15, 2016
Ido Flatow
Production Debugging Web Applications

2. THE STORIES YOU ARE ABOUT TO HEAR
ARE BASED ON ACTUAL CASES.
LOCATIONS, TIMELINES, AND NAMES
HAVE BEEN CHANGED FOR DRAMATIC
PURPOSES AND TO PROTECT THOSE
INDIVIDUALS WHO ARE STILL LIVING.

3. For the Next 60 Minutes…
Introduction
Service hangs
Unexplained exceptions
High memory consumption

4. Why Are You Here?
You are going to hear about
Bugs in web applications
Tips for better coding
Debugging tools, and when to use them
You will not leave here as expert debuggers! Sorry
But… You will leave with a good starting point
And probably anxious to check your code

5. How Are we Going to Do This?
What did the client report?
Which steps we used to troubleshoot the issue?
What did we find?
How did we fix it?
What were those tools we used?

6. The Tired WCF Service
Client
Local bank
Reported
WCF service works fine for few hours, then stops handling requests
Clients call the service, wait, then time out
Server CPU is high
Workaround
Restart IIS Application pool

7. Troubleshooting
Configured WCF to output performance counters
Used Performance Monitor to
watch WCF’s counters, specifically
Instances
Percent Of Max Concurrent Calls

8. Troubleshooting - cntd
Waited for the service to hang
Inspected counter values
Value was at 100% (101.563% to be exact)
At this point, no clients were active!
Reminder - WCF throttles concurrent calls (16 x #Cores)

9. Troubleshooting - cntd
Watched w3wp thread stacks
with Process Explorer
Noticed many .NET threads in sleep loop
Issue found - Requests hanged in the service, causing it to
throttle new requests
Fixed code to stop endless loop – problem solved!

10. The Tools in Use
Performance Monitor (perfmon.exe)
View counters that show the state of various application aspects
Most people use it to check CPU, memory, disk, and network state
.NET CLR has useful counters for memory, GC, JIT, locks, threads, exceptions,
etc.
Other useful counters: WCF, ASP.NET, IIS, and database providers
Sysinternals Process Explorer
Alternative to Task Manager
Select a process and view its managed and native threads and stacks
Examine each thread’s CPU utilization
View .NET CLR performance counters per process
https://download.sysinternals.com/files/ProcessExplorer.zip

11. Why We Do Volume Tests
Client
QA team. Government collaboration app
Reported
MVC web application works in regular day-to-day use
Application succeeded under load tests
Under volume tests, application throws unexplained errors
Returns HTTP 500, with no specific error message
Application logs are not showing any relevant information
Workaround
None. Failed under volume tests

12. Troubleshooting
Checked Event Viewer for errors, found nothing
Used Fiddler to view the HTTP 500 response
Error text was too general, not very useful

13. Troubleshooting - cntd
Decided to use IIS Failed Request Tracing
Luckily, the MVC app had an exception filter that used tracing
Created a Failed Request Tracing rule for HTTP 500
Added the System.Web.IisTraceListener to the web.config
Waited for the test to reach its breaking point…

14. Troubleshooting - cntd
Opened the newly created trace file in IE
Found an error! Exception in JSON serialization - string too big
Stack overflow to
the rescue…

15. Troubleshooting - cntd
Ran the test again – failed again!
Checked the JavaScriptSerializer serialization code
Where is MaxJsonLength set?
Inspected MVC’s JsonResult code
Found the code that configured the serializer

16. Troubleshooting – almost done
Code fix was quite easy
But how big was our JSON string? 5MB? 1GB?
Time to grab a memory dump…
return Json(data); return new JsonResult {
Data = data,
MaxJsonLength =
};

17. Troubleshooting – just one more thing
Quickest way to dump on an exception - DebugDiag

18. Troubleshooting – final piece of the puzzle
Tricky part, using WinDbg to find the values

19. Troubleshooting – final piece of the puzzle
Which thread had the exception - !Threads

20. Troubleshooting – final piece of the puzzle
Get the thread’s call stack - !ClrStack
JavaScriptSerializer.Serialize takes a StringBuilder …

21. Troubleshooting – final piece of the puzzle
List objects in the stack - !DumpStackObjects (!dso)

22. Troubleshooting – final piece of the puzzle
Get the object’s fields and values - !DumpObj (!do)

23. The Tools in Use
Fiddler
HTTP(S) proxy and web debugger
Inspect, create, and manipulate HTTP(S) traffic
View message content according to its type, such as image, XML/JSON, and JS
Record traffic, save for later inspection, or export as web tests
http://www.fiddlertool.com
IIS Failed Request Tracing
Troubleshoot request/response processing failures
Collects traces from IIS modules, ASP.NET pipeline, and your own trace
messages
Writes each HTTP context’s trace messages to a separate file
Create trace file on: status code, execution time, event severity
http://www.iis.net/learn/troubleshoot/using-failed-request-tracing

24. The Tools in Use
Decompilers
Browse content of .NET assemblies (.dll and .exe)
Decompile IL to C# or VB
Find usage of a field/method/property
Some tools support extensions and Visual Studio integration
http://ilspy.net
https://www.jetbrains.com/decompiler
http://www.telerik.com/products/decompiler.aspx

25. The Tools in Use
DebugDiag
Memory dump collector and analyzer
Can generate stack trees, mini dumps, and full dumps
Automatic dump on crash, hanged requests, perf. counter triggers, etc.
Contains an analysis tool that scans dump files for known issues
https://www.microsoft.com/en-us/download/details.aspx?id=49924
WinDbg
Managed and native debugger, for processes and memory dumps
Shows lists of threads, stack trees, and stack memory
Query the managed heap(s), object content, and GC roots
Various extensions to view HTTP requests, detect dead-locks, etc.
https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk

26. Leaking Memory In .NET – It Is Possible!
Client
Local insurance company
Reported
Worker process memory usage increase over time
Not sure if it’s a managed or a native issue
Workaround
Increase application pool recycle to twice a day

27. Troubleshooting
First, need to know if the leak is native or managed
Checked process memory with Sysinternals VMMap
Looking at multiple snapshots, seems to be managed (.NET) related

28. Troubleshooting - cntd
Time to get some memory dumps
Need several dumps, so we can compare them
Very simple to do, using Windows Task Manager
Next, open them and compare memory heaps

29. Troubleshooting - cntd
Compared the dumps with Visual Studio 2015
(Requires the Enterprise edition)

30. Troubleshooting - cntd
Didn’t take long to notice the culprit and reason
Hundreds of DimutFile objects, each containing large byte arrays

31. Troubleshooting - cntd
These objects were not “leaked”, they were cached!
Recommended fix included
Do not cache many large objects
Cache using an expiration (sliding / fixed)

32. Troubleshooting – wait a second…
The memory diff. had another suspicious leak
Why are we leaking the HomeController?

33. Troubleshooting - cntd
Checked roots
Controller is also cached, why?
Referenced by the CacheItemRemovedCallback event

34. Troubleshooting - cntd
Checked the code for last time
CacheItemRemoved is registered to the event, but it is an instance
method
Note - adding instance method to a global event may leak its containing
object
The fix - change the callback method to static

35. The Tools in Use
Sysinternals VMMap
Helps in understanding and optimizing memory usage
Shows a breakdown of the process memory types
Displays virtual and physical memory
Can show a detailed memory map of address spaces and usage
https://technet.microsoft.com/en-us/sysinternals/vmmap.aspx
Visual Studio managed memory debug (Enterprise)
Part of Visual Studio’s dump debugger
Displays list of object types and their inclusive/exclusive sizes
Tracks each object’s root paths
Compare memory heaps between dump files
https://msdn.microsoft.com/en-us/library/dn342825.aspx

36. When SSL/TLS Fails…
Client
Airport shuttle service site
Reported
Application suddenly fails to communicate with external services over
HTTPS
Error is “Could not establish trust relationship for the SSL/TLS secure
channel”
Cannot reproduce the error in dev/test
Workaround
Restart IIS (iisreset.exe )

37. Troubleshooting
Checked Event Viewer for any related error
Found the SSL/TLS error in the Application and System logs
According to MSDN documentation, error code 70 is protocol version support

38. Troubleshooting - cntd
Used Microsoft Message Analyzer (network sniffer) to watch the
TLS handshake messages
Before issue starts – client asks for TLS 1.0, handshake completes
After issue starts – client asks for TLS 1.2, handshake stops

39. Troubleshooting - cntd
Checked the Server Hello, it returned TLS 1.1, not 1.2
Switched to TCP view to verify client’s behavior
Client indeed sends a FIN, and server responds with an RST

40. Troubleshooting – moment of clarity
Developer remembered adding code to support new Paypal
standards of using only TLS 1.2
Code set to only use TLS 1.2, removing support for TLS 1.0 and 1.1
Suggested fix
Use enum flags to support all TLS versions – Tls | Tls11 | Tls12
This is the actual default for .NET 4.6 and on
For .NET 4.5.2 and below – default is Ssl3 | Tls

41. The Tools in Use
Microsoft Message Analyzer
Replaces Microsoft’s Network Monitor (NetMon)
Captures, displays, and analyzes network traffic
Can listen on local/remote NICs, loopback, Bluetooth, and USB
Supports capturing HTTPS pre-encryption, using Fiddler proxy
component
https://www.microsoft.com/en-us/download/details.aspx?id=44226
Event Viewer (eventvwr.exe)
Discussed previously

42. Additional Tools (for next time…)
Process monitoring
IIS Request Monitoring, Sysinternals Process Monitor
Tracing and logs
PerfView (CLR/ASP.NET/IIS ETW tracing), IIS/HTTP.sys logs, IIS Advanced
Logging, Log Parser Studio
Dumps
Sysinternals ProcDump, DebugDiag Analysis
Network sniffers
Wireshark

43. How to Start?
Understand what is happening
Be able to reproduce the problem ”on-demand”
Choose the right tool for the task
When in doubt – get a memory dump!

44. Resources
You had them throughout the slides 
My Info
@IdoFlatow // idof@sela.co.il //
http://www.idoflatow.net/downloads