Mais conteúdo relacionado
Semelhante a Windows Debugging Tools - JavaOne 2013 (20)
Windows Debugging Tools - JavaOne 2013
- 1. Matthew Kilner – IBM Java L3 Service – Core team lead
23rd September 2013
Windows Debugging Tools
© 2013 IBM Corporation
- 2. Important Disclaimers
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION
CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED.
ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED
ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR
INFRASTRUCTURE DIFFERENCES.
ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.
IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT
PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.
IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE
USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
- CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR
SUPPLIERS AND/OR LICENSORS
2
© 2013 IBM Corporation
- 3. About me
Matthew Kilner
Work for IBM
– 13 years working on IBM Java
• Memory Management
• Class Sharing
• RAS
– Currently leading the Core customer support team.
Contact info
– kilnerm@uk.ibm.com
– Twitter: @IBMJTC
– Youtube: IBM_JTC
Visit the IBM booth #5112 and meet other IBM developers
at JavaOne 2013
3
© 2013 IBM Corporation
- 4. What should you get from this talk?
Awareness of some excellent windows tools:
– A high level understanding of what it is they let you do,
– Details of where to obtain the tools,
– Details of where to find out how to use them.
–
Some useful tips on working with the tools
© 2013 IBM Corporation
- 5. General tips
Most Microsoft tools require symbol files, often known as .pdb, to enable method names and source
lines in stack traces.
Always keep your symbols in one place and keep a pre-configured Symbol path to hand.
• I will use generic paths for the JDK symbol files and overlay those paths with the correct level of
symbol for the JDK I am debugging at the time.
• JDK symbol files ship in the majority of the IBM JDK packaging formats.
Remember to include the MS symbol server:
• SRV*<local symbol store>*http://msdl.microsoft.com/download/symbols
5
© 2013 IBM Corporation
- 6. PDB files
PDB files are an essential part of effective debugging
The following link provides deailed information on how to generate them:
• http://msdn.microsoft.com/en-us/library/aa260783%28v=vs.60%29.aspx
Requires the addition of compile and link time flags.
6
© 2013 IBM Corporation
- 7. IBM Presentation Template Full Version
The tools
Windows SDK tools:
– UMDH
– ADPlus
Sysinternals tools:
– VMMAP
– Procdump
Standard tools:
– Performance Monitor
Stand alone tools:
– Debug Diagnostics
7
© 2013 IBM Corporation
- 8. Usefulness is contextual
A tools usefulness is limited to a number of problem types or tasks you are working on.
Most of these tools have been “discovered” during analysis of real world problems.
– Predominantly related to Virtual addressing problems.
–
Several of the tools have capabilities beyond the use described today.
8
© 2013 IBM Corporation
- 9. Monitoring the system / application
There are two common problem scenarios where it is desirable to understand more about how a
system or application is performing:
– Virtual Addressing problems
– Performance problems
• Slow responsiveness, High CPU usage, etc....
9
© 2013 IBM Corporation
- 10. Performance Monitor
Where to get it:
• Included in you windows installation
What is it:
–A powerful monitoring tool
What does it let us do:
–Monitor and analyse performance data in real time or from a log file.
–Can track almost every aspect of system performance.
How do you use it:
–Launched by:
• Entering “perfmon” in your run dialog, or
• Via the Performance Monitor entry on the start menu
–Read the in-built help documentation.
10
© 2013 IBM Corporation
- 11. Performance Monitor – monitoring process size
The most common use is for
accurate measurement of a
processes size while debugging
virtual addressing issues.
Define a data collector Set to record
the following counters on the process
object:
• Virtual bytes
• Private bytes
•
11
© 2013 IBM Corporation
- 12. Performance Monitor – monitoring process size
For live monitoring add the same
counters to the performance monitor
view
When loading Data collection sets
back into the tool you will see similar
output.
•
12
© 2013 IBM Corporation
- 13. Performance Monitor – other uses
We primarily use it for monitoring process
sizes which only scratches the surface of
its practical uses.
The ability to monitor nearly anything you
want on one or more processes such as:
• Monitoring per thread CPU
consumption across a process.
• Monitoring the power consumption of
individual CPU's.
13
© 2013 IBM Corporation
- 14. Capturing Dumps
Occasionally we see problems where the JDK does not write dumps when we expect it to.
– This provides a real challenge for continuing problem analysis.
–
We use two tools that help capture dumps in this scenario:
– ADPlus (Windows SDK)
– Procmon (Sysinternals)
–
14
© 2013 IBM Corporation
- 15. ADPlus
Where to get it:
• http://msdn.microsoft.com/en-us/library/windows/desktop/hh852363.aspx
• Part of the Windows SDK
What is it:
– A useful tool for generating system dumps
– Can also be used in hang situations
What does it let us do:
– Monitor a process and generate system dumps on:
• User request
• Process termination
• First and second chance exceptions.
How do you use it:
– Documentation is included in the install directory
• <Path to installation>Windows Kits8.0Debuggersx64ADPlus.doc
15
© 2013 IBM Corporation
- 16. Procdump
Where to get it:
• http://technet.microsoft.com/en-us/sysinternals/dd996900
What is it:
–A useful tool for generating system dumps
What does it let us do:
–Generate system dumps in a multitude of scenarios:
• The process exceeding a threshold on any performance counter
• On process exit
• On first and second chance exceptions.
How do you use it:
–Command line utility
• See the link above for examples, or
• Run the tool with no options.
16
© 2013 IBM Corporation
- 17. Procdump – usage examples
Example: Write up to 3 dumps of a process named 'consume' when it exceeds 20% CPU usage for
five seconds to the directory c:dumpconsume with the name consume.dmp:
• C:>procdump -c 20 -s 5 -n 3 -o consume c:dumpconsume
Example: Write 3 dumps 5 seconds apart:
• C:>procdump -s 5 -n 3 notepad.exe notepad.dmp
Example: Launch a process and then monitor it for excessive CPU usage:
• C:>procdump -c 30 -s 10 -x consume.exe consume.dmp
17
© 2013 IBM Corporation
- 18. Understanding Virtual Memory Growth
A common problem we encounter is the virtual address space of processes growing and either:
– Causing process termination due to an inability to allocate memory, or
– Causing severe performance issues due to paging
There are three excellent tools available on Windows to help understand what is causing the growth:
– UMDH (Windows SDK)
– Debug Diag ( Stand alone)
– VMMAP (Sysinternals)
18
© 2013 IBM Corporation
- 19. UMDH
Where to get it:
• http://msdn.microsoft.com/en-us/library/windows/desktop/hh852363.aspx
• Part of the Windows SDK
What is it:
–A tool that tracks and analyzes windows heap allocations within a process
What does it let us do:
– Take snapshots of memory footprint within a process broken down by allocation stack.
– Compare snapshots and summarize the key differences.
– Identify stacks for specific allocation sizes.
How do you use it:
– Basic usage instructions:
• http://msdn.microsoft.com/en-us/library/windows/hardware/ff558947%28v=vs.85%29.aspx
– More detail is available in this Microsoft knowledge base article:
• http://support.microsoft.com/kb/268343
–
19
© 2013 IBM Corporation
- 20. UMDH – sample output
// Connecting to process 3312 ...
// Process 3312 opened handle=48.
// Loaded modules:
Snapshot files contain the raw data
from a point of execution of the
process, you will see:
– A summary of loaded modules
– A summary fo the data format:
– A series of allocations:
–
When you have multiple snapshots
you run the compare tool.
//
Base Size Module
//
13A0000
//
773B0000 1A9000 C:windowsSYSTEM32ntdll.dll
35000 C:Program FilesIBMCodeRallyjdkjrebinjavaw.exe
// Process modules enumerated.
// _NT_SYMBOL_PATH set by default to C:windowssymbols
// Debug library initialized ...
DBGHELP: ntdll - export symbols
*- - - - - - - - - - Start of data for heap @ 310000 - - - - - - - - - REQUESTED bytes + OVERHEAD at ADDRESS by BackTraceID
STACK if not already dumped.
*- - - - - - - - - - Heap 310000 Hogs - - - - - - - - - 28 bytes + 28 at 311310 by BackTrace720C0
7744BBED
773DC2E2
773F357E
773F1937
773DC34E
20
© 2013 IBM Corporation
- 21. UMDH – comparing snapshots
To run a comparison you execute:
• UMDH <log 1> <log 2> > <output file>
Once the compare completes the output file contains a list of the call stacks that have differing memory
usage between the dumps sorted by overall size:
–
+ 550376 ( 550776 - 400) 12 allocsBackTrace33599F60
+
11 ( 12 1) BackTrace33599F60
allocations
ntdll!MD5Final+A63D
J9PRT26!j9mem_allocate_memory+74 (c:cygwinhomeforemansandboxjvm-binsrcj9portcommonj9memtag.c, 184)
j9vm26!allocateMemoryForSegment+A0 (c:cygwinhomeforemansandboxjvm-binsrcj9vmsegment.c, 216)
j9vm26!allocateVirtualMemorySegmentInListInternal+E6 (c:cygwinhomeforemansandboxjvm-binsrcj9vmsegment.c, 357)
j9vm26!allocateFixedMemorySegmentInList+A8 (c:cygwinhomeforemansandboxjvm-binsrcj9vmsegment.c, 319)
21
© 2013 IBM Corporation
- 22. UMDH – important considerations
The tool uses snapshots of outstanding allocations so a degree of additional interpretation is needed.
Do not assume the top consumer between two snapshots is the cause of the problem
• Also consider the volume of allocations, particularly for small chunks which have implications in
terms of fragmentation pre Vista/Server 2008.
•
Best used in conjunction with performance monitor and over several iterations of snapshots.
Longer durations between snapshots often make it easier to identify leaks
22
© 2013 IBM Corporation
- 23. Debug Diagnostic Tool
Where to get it:
• http://www.microsoft.com/en-us/download/details.aspx?id=26798
What is it:
–A tool providing a range of capability for analysis of crash, hang or native memory isssues.
What does it let us do:
–Analysis of native memory issues.
–Basic analysis of crash or hang scenarios.
–Attaches to live processes and can be used to generate dumps at request or by rule configuration.
–Designed for minimal performance impact
How do you use it:
– Basic Instructions:
• http://support.microsoft.com/kb/2580960
–Read the whitepaper:
• http://www.microsoft.com/download/en/details.aspx?id=23521
–Use the in-built help documentation.
23
© 2013 IBM Corporation
- 24. Debug Diagnostic Tool – native memory analysis
Our main use of this tool is for debuging native memory issues.
–The key thing to remember is to always inject the leak track dll, without this the reports you generate
will be basic.
–
–The optimal method of analysis involves taking multple dumps of the process as its virtual size
grows, understanding the rate of growth between dumps is important and can be monitored using
the performance monitor tool.
–
–Dumps can be generated directly from the UI or rules can be configured to generate dumps after a
set amount of time or increase in memory usage.
–
–Once dumps are generated you load them into the tool and select to run the Memory pressure
script. When complete the tool will open the report in a browser window.
–
24
© 2013 IBM Corporation
- 25. Debug Diagnostic Tool – native memory report
The report is broken into distinct sections:
• Virtual Memory Analysis Report
• Heap Analysis Report
• Leak Analysis Report (only with leak tracking)
•
The following extracts are taken from results of monitoring a 64 bit process which has a maximum
address space of 8 Tbytes.
Each report starts with a summary which contains some key pieces of information:
25
© 2013 IBM Corporation
- 26. Debug Diagnostic Tool – Virtual Memory Analysis Report
Contains:
• Virtual Memory Summary
• Virtual Memory Details
• Virtual Allocation Summary
• Loaded Module Summary
• Thread Summary
The most pertinent details are the first
three elements
•
26
© 2013 IBM Corporation
- 27. Debug Diagnostic Tool – Heap Analysis Report
The heap analysis report
contains information on
the Heaps in use within
the process.
Not to be confused with
the JVM's heap.
27
© 2013 IBM Corporation
- 28. Debug Diagnostic Tool – Heap Details
For each heap you get:
– A summary
– Information on the Heap segments
– Information on allocations
The Summary:
28
© 2013 IBM Corporation
- 31. Debug Diagnostic Tool – Leak Analysis Report
Only available when the leaktrack dll is injected to the process being monitored.
The most useful aspect is the summary of allocations at the module level:
– By Count:
–
31
© 2013 IBM Corporation
- 33. Debug Diagnostic Tool – Module Summary
For each module we get a more detailed breakdown including a summary:
33
© 2013 IBM Corporation
- 34. Debug Diagnostic Tool – Module Summary
We get a summary of functions by count and allocated size:
34
© 2013 IBM Corporation
- 35. Debug Diagnostic Tool – Module Summary
And for each function additional detail:
35
© 2013 IBM Corporation
- 36. Debug Diagnostic Tool – Module Summary
Allocation count and size breakdown:
36
© 2013 IBM Corporation
- 37. Debug Diagnostic Tool – Module Summary
And sample call stacks for prominent sizes:
37
© 2013 IBM Corporation
- 38. Debug Diagnostic Tool – Drawing conclusions
The tool provides a lot of information, but doesn't do comparison of multiple data sets.
Using a spreadsheet to capture the pertinent data points highlighted over the previous slides will build
a picture that allow you to see the key areas that are causing growth.
– You can even plot the data for easier visualization
38
© 2013 IBM Corporation
- 39. VMMAP
Where to get it:
• http://technet.microsoft.com/en-us/sysinternals/dd535533
What is it:
–A tool for analysis the virtual and physical memory usage of a windows process
What does it let us do:
–Attaches to live processes for run time analysis and allows for export of process information in a
variety of formats.
–Provides a facility to launch a process and trace it which utilizes additional instrumentation that
enables us to identify the stacks allocating into specific regions of memory.
–Allows us to compare the contents of the address space between two different points in time, when
tracing is enabled you can see exactly who is allocating into regions.
–At any time you can flush the physical memory the process is using meaning this is a great tool for
seeing just how much memory a particular feature of your application uses.
–
–
39
© 2013 IBM Corporation
- 41. VMMAP – Comparing snapshots
First you have to open the timeline window:
Next you left click and drag to select the
timeframe you wish to view:
• Snapshots are represented by the
small black lines:
41
© 2013 IBM Corporation
- 42. VMMAP – Comparing snapshots
Once selected
you will see:
42
© 2013 IBM Corporation
- 43. VMMAP – Comparing snapshots
Address ranges that are in the most recent snapshot but not in the earliest are highlighted in bright
green.
Address ranges that were in the earliest snapshot but not in the current are highlighted in bright red.
If you started the process under full tracing you can select a “HEAP” and view the allocations made
within it between the snapshots, for each allocation you can also view the call stack.
43
© 2013 IBM Corporation
- 44. VMMAP – other features
When tracing is enabled you have the option to view a trace of key memory operations by selecting the
trace option which lists:
• Allocation sites
• Number of allocations for the site
• The number of bytes allocated
• The type of operation
•
You can also select any site and choose
to view its stack.
44
© 2013 IBM Corporation