A techis guide to combating bugs & poor performance in production
•
0 likes•1,866 views
"Session Presentation from Visual Studio 2012 Launch" A Techie without efficient tools is only half the good! This session will give you the armor to battle the 2 most common scenarios we run into. 1. A production defect that cannot be reproduced in a test environment. 2. Not being able to reproduce poor application performance experienced in Production in a test environment. This session targets to show you how to use IntelliTrace in Production and Visual Studio Standalone profiler to the best of your advantage.
Recommended
Recommended
More Related Content
Similar to A techis guide to combating bugs & poor performance in production
Similar to A techis guide to combating bugs & poor performance in production (20)
Recently uploaded
Recently uploaded (20)
A techis guide to combating bugs & poor performance in production
- 1. Microsoft IT (India) – Visual Studio 2012 Launch 3/October to 5/October
- 2. Tools to combat Bugs & Poor Performance in Production Tarun Arora, Microsoft MVP Microsoft IT India
- 3. Cmd>Who am I?
- 4. Today’s Menu Card How to tackle non-reproducible production defects? > IntelliTrace in Production How to approach production performance issues? > Visual Studio Standalone Profiler How to get your boss to love you? > Automating performance test creation
- 5. Disclaimer
- 6. What developers love <3 Building a Solid Architecture Fixing testing defects Fixing production issues
- 7. Why are Production defects hard to fix?
- 8. Spot the real Problem?
- 9. Team Barriers = Value Delivery Impediments ||
- 10. Production Incidents Hard to Debug and Resolve Production errors Difficult to identify root cause, debug code and resolve defects Problem Actionable diagnostics IntelliTrace in production to speed debugging and code fix Solution
- 13. Recap – IntelliTrace in Production • IntelliTrace collector supports an Xcopy setup • P in PDBs stand for “Priceless” • iTrace opens only in Visual Studio Ultimate • CTP 1 brings Custom Events and TimeStamp • IntelliTrace is not just for production. You can leverage it in Visual Studio, Microsoft Test Manager and most importantly via System Center.
- 14. Today’s Menu Card How to tackle non-reproducible production defects? > IntelliTrace in Production How to approach production performance issues? > Visual Studio Standalone Profiler How to get your boss to love you? > Automating performance test creation
- 15. Poor performance in Prod – what’s that? “The progress bar in IE is moving but nothing is happening on the page!”
- 16. How are performance issues diagnosed? Framework Problem Resolution Define – Analyse – Refine Problem Statement Each tool has a different level of intrusion Flexibility to disruption of service in Production
- 17. Apples VS Oranges Sampling Trace (Instrumentation) Statistical Profiling Analytical Profiling Code being Profiled Profiler FuncFoo1( ) FuncFoo1 Enter Probe … External function enter Probe Console.WriteLine(“Helllo”); Process A Idle Process B Process A External function exit Probe Process A Process A - Func1 - Func3 - Func1 - Func2 …. - Func2 - Func1 - Func2 - Func1 - Func3 - Func2 - Func3 - Func3 FuncFoo2( ) - Stack - Stack - Stack - Stack … FuncFoo2 Exit Probe Exit FuncFoo1()
- 18. Visual Studio Standalone Profiler
- 20. Memory Management - Insight
- 21. Standalone Profiler – Commands Vsperfreport /packsymbol Vsperfreport /packsymbols /summary: all <myTraceFile>.vsp /summary:all <mySamplefiile.vsp> Vsinstr /ExcludeSmallFuncs /NoWarn:2001;200 Sn – Vr *.* FileToBeInstrumented.dl Vsperfclrenv /globaltraceon Vsperfclrenv /globalsampleon Vsperfcmd /start:Trace /output:<mytracefile> Vsperfcmd /start:sample /output:<sampleFile> VSPerfAspNetCmd /tip http://foo:8085/ /cs /globaloff /user:<domain><user1> /attach:<PID> /o:<myFileName>.vsp Vsperfcmd /detach Vsperfcmd /status Vsperfcmd /shutdown Vsperfcmd /globalon Vsperfcmd /globaloff Vsperfcmd /shutdown VsPerfMon </Trace [/Counter:cfg] | /Sample | /Coverage> /CrossSection /Output <file name> [WinCounter:cfg] [/User [Domain]username]
- 23. Recap – Visual Studio Standalone Profiler • How to setup Visual Studio Standalone collector • Sampling Vs Instrumentation • When to use VsPerfCmd and when to use VsPerfAspNetCmd • Sampling does not support TIP on Win 8/2012 • TIP requires Visual Studio Ultimate
- 24. Today’s Menu Card How to tackle non-reproducible production defects? > IntelliTrace in Production How to approach production performance issues? > Visual Studio Standalone Profiler How to get your boss to love you? > Automating performance test creation
- 25. Automation Everywhere! “Speed up performance issue resolution by automating web tests using IIS Logs”
- 26. public IEnumerable<WebTestRequest> GetRequests() { LogQuery logQuery = new LogQuery(); IISLogInputFormat iisInputFormat = new IISLogInputFormat(); // currently these columns give us suffient information to construct the web test requests string query = @"SELECT s-ip, s-port, cs-method, cs-uri-stem, cs-uri-query FROM " + _iisLogPath; LogRecordSet recordSet = logQuery.Execute(query, iisInputFormat); // Apply a bit of transformation while (!recordSet.atEnd()) { ILogRecord record = recordSet.getRecord(); if (record.getValueEx("cs-method").ToString() == "GET") { string server = record.getValueEx("s-ip").ToString(); string path = record.getValueEx("cs-uri-stem").ToString(); string querystring = record.getValueEx("cs-uri-query").ToString(); StringBuilder urlBuilder = new StringBuilder(); urlBuilder.Append("http://"); urlBuilder.Append(server); urlBuilder.Append(path); if (!String.IsNullOrEmpty(querystring)) { urlBuilder.Append("?"); urlBuilder.Append(querystring); } // You could make substitutions by introducing parameterized web tests. WebTestRequest request = new WebTestRequest(urlBuilder.ToString()); yield return request; } ...
- 27. Demo Web performance tests from IIS Logs
- 28. Summary How to tackle non-reproducible production defects? > IntelliTrace in Production How to approach production performance issues? > Visual Studio Standalone Profiler How to get your boss to love you? > Automating performance test creation
Editor's Notes
- Tarun Arora is a passionate technologist working in Avanade, London. In his day job Tarun helps leading Energy Trading and Financial companies develop and support efficient, scalable and robust enterprise applications. His specialisms include Application Lifecycle Management (ALM) with Microsoft development tools with a strong expertise in implementing & leading Waterfall and Agile based development processes supporting distributed teams on large-scale .NET based enterprise projects. Tarun is a Microsoft MVP and a Visual Studio ALM Ranger. Outside of work he is found moderating MSDN forums, blogging and writing for MSDN, TechNet, Codeproject and other technology focused websites. For latest and greatest follow him on @arora_tarun
- Today I am not going to try and solve all your problems but inspire you with real life examples to go back and try this stuff out.
- -> Important to highlight that all the commands and demos are web applications focussed. Some of this stuff does not work for windows applications.
- Sarcastic!If there is an easy road out, chances are developers will walk that path. 1. Building a solid architecture – At the start of the projects, the developers are inspired to implement the best and proven patterns but things change under stringent timelines.2. Fixing Testing Defects – Any testers in the audience. Did I get this right? There was actually a very funny cartoon in the blogosphere the other day around an excuse matrix developers present when faced with a defect by the QA team. How often have you seen developers close defects simply saying, you are doing something wrong, its suppose to work this way, it just works on my machine. 3. Fixing Production issues – Production issues that are hard to reproduce usually never see the light of a fix and are closed with the remarks “Not reproducible”Why this behaviour?
- A small survey with clients revealed very interesting numbers.
- Where is the realroot of the problem?The application hangs or simply crashes. How does the team usually investigate? Event Logs, Application Logs, IIS logs are a good start. Application Stack trace in case of an exception could be helpful. Some applications make use of System.Diagnostics to write the application activity to a log file. Frameworks such as log4net, Nlog, Enterprise Library provide similar functionality. It can sometimes be helpful to up the level of verbosity and see the log files. This can sometimes work in case of simple issues, but if you are dealing with complex financial models churning large loads of external data then it becomes close to impossible to reproduce the issue. This results in infinite cycles between the development team and the dev ops team. Data Security applies in certain organizations, it is not permitted to look at the system data in live system.Most Common Ways to diagnose Production IssuesLoggingInstrumentationMemory DumpsToooo many iterations, too longer cycle time, really kills the drive to solve the bug simply because it becomes too costly to reproduce it.
- Software development in enterprises and business organizations is a cross-functional undertaking. Barriers and chasms between the cross-functional teams that need to integrate in delivering software are the top impediments that hinder value delivery. Such impediments are usually a manifestation of rigid processes, ineffective collaboration tools, and development practices that do not take advantage of the advances in technology and opportunities to better integrate teams.
- As before, once bugs in production are discovered, information must flow from the Operations team back to the Development team. A new feature to dramatically reduce Mean Time to Repair is the new ability to run IntelliTrace in production. IntelliTrace in production is easy to run, and collects critical IntelliTrace logs with minimal impact to server performance. Developers, who are already familiar with using this data in test environments, now have the data to speed root cause analysis of production bugs, and rapidly identify the needed code fix.Another important integration is the use of translatable artifacts. For instance, System Center AVICode logs (from operations) will be directly convertible to IntelliTrace logs in System Center 2012 SP1. Since ops personnel are familiar with AVICode logs, they can gather data in a familiar format, while still providing deep troubleshooting information to developers in a form that they can understand and act on.
- So… What is IntelliTrace?Take the analogy of a plane crash (Blackbox) to explain Historical Debugging. When a plane crashes we want to know why the crash happenedTrying to reproduce the crash is costly and can take a lot of timeHow can we find out what happened without crashing new planesWhere can IntelliTrace be applied, Visual Studio – F5 debugMicrosoft Test Manager – While executing test casesProduction – Stand alone IntelliTrace collectorCloud – Windows AzureTalk about the license requirements – Visual Studio Ultimate to view the results collected by the stand alone collector. The stand alone collector does not require any license, it is free to be used. You can run the IntelliTrace collector on any machine that has > .net framework 2.0 and any machine with IIS 7.0 and above will have this already.
- DemoShow IntelliTrace collector and a simple walkthrough of IntelliTrace in productionShow pdb information in dll metadata along with source indexationTalk about – Symbol Server and Source Indexation Discuss Custom Events and iTrace summaryDiscuss remote IntelliTrace kick offDiscuss how as a consultant I benefitted – success story
- RecapShout out loud what is not possible to do with IntelliTrace but what your feedback can help shape as a future requirementIt is not possible to profile Windows ServiceIntelliTrace output file bloats if you run it over recursive statements and the results might not be very helpfulIntelliTrace run will force an application pool restart of your web applicationCustom events do not support overloaded methodsIntelliTrace does not record variables to stored procedures or the results. This is because some organizations have data security.
- Alright, let’s move on to part 2“How to approach production performance Issues”
- “The application is too slow!” “The application is slower than yesterday!” “The progress bar in IE is moving but nothing is happening on the page!”Did performance problems happen overnight, or did they sneak up on you?Where it all begins? No performance requirements at project levelNo benchmarks on Tx/s, concurrent users, pages/seconds, etc.
- Any production issue can be classified under 3 bucketsHangCrashPerformanceThe first 2 can be addressed using IntelliTrace in ProductionUnderstanding and troubleshooting performance problems in complex software systems is notoriously challenging. This challenge is compounded for software in production for several reasons. To avoid slowing down production systems, analysis and troubleshooting must incur minimal overhead. Further, performance issues in production can be both rare and non-deterministic, making the issues hard to reproduce. The Problem Resolution framework works best when troubleshooting performance issues. It’s important to articulate the issue in a Problem Statement, for example “The Web application tends to hang when the run button is clicked on the Employee page”. Further analysis will give you a lead into what happens under the hood when the run button is clicked, this is a great opportunity to review the problem statement again. I have found great success in not believing everything the user tells me. Any tools run to diagnose the issues in production incur a cost in the shape of performance hit. Not all customers are ready to take that plunge. So, it is important to classify the possible tools and techniques into 2 buckets, low impact and high impact. Low impact tools have a low cost or minor affect, meaning, the end user can continue to use production without any down time. Low Impact Tools,Debug Diag – Debug diag can be used to generate a dump. It usually takes 20-30 seconds to create a memory dump. Be aware that the application processing is interupted during this time. The greater the memory consumption of the process, the more time it takes to generate the dump. In those circumstances a mini dump can be a possibility.Asp.net health monitor can be a useful tool too.Perf Mon – I sometimes feel this is so under utilized. I have seen many support teams using Task Manager to track memory usage and process memory over per mon. For starters perfmon has a history view so at least its possible to analyse the cpu utilization trend. Perfmon has great counters for diagnosing ASP.net application performance issues. There is a very interesting counter to see no of exceptions/sec. Usually support teams tend to simply ignore it. Fixing exceptions is one of those low hanging frutts. Anything over 15 exceptions/sec can start showing great impacts on the performance of your application. High Impact Tools,Managed Stack Explorer – Though some of my peers count it as a low impact tool, but I have had managed stack explorer crash my production application. So, it is not usually in my first few list of tools when diagnosing production issues.Live Debugger – Nothing better than live debugging. But it is important to know that this is absolutely at the extreme end of the impact spectrum. Everytime a live debugger is hit the application execution is interrupted. I would only recommend using this in Pre-Production. Profilers – Profilers are a great tool to narrow down to the root cause of the problem. Performance problems are usually a result of, Poor memory management which could also lead to high CPU utilizationA result of network lag or I/OConcurrency Which is where profilers excel :-)
- Any production issue can be classified under 3 bucketsHangCrashPerformanceThe first 2 can be addressed using IntelliTrace in ProductionUnderstanding and troubleshooting performance problems in complex software systems is notoriously challenging. This challenge is compounded for software in production for several reasons. To avoid slowing down production systems, analysis and troubleshooting must incur minimal overhead. Further, performance issues in production can be both rare and non-deterministic, making the issues hard to reproduce. The Problem Resolution framework works best when troubleshooting performance issues. It’s important to articulate the issue in a Problem Statement, for example “The Web application tends to hang when the run button is clicked on the Employee page”. Further analysis will give you a lead into what happens under the hood when the run button is clicked, this is a great opportunity to review the problem statement again. I have found great success in not believing everything the user tells me. Any tools run to diagnose the issues in production incur a cost in the shape of performance hit. Not all customers are ready to take that plunge. So, it is important to classify the possible tools and techniques into 2 buckets, low impact and high impact. Low impact tools have a low cost or minor affect, meaning, the end user can continue to use production without any down time. Low Impact Tools,Debug Diag – Debug diag can be used to generate a dump. It usually takes 20-30 seconds to create a memory dump. Be aware that the application processing is interupted during this time. The greater the memory consumption of the process, the more time it takes to generate the dump. In those circumstances a mini dump can be a possibility.Asp.net health monitor can be a useful tool too.Perf Mon – I sometimes feel this is so under utilized. I have seen many support teams using Task Manager to track memory usage and process memory over per mon. For starters perfmon has a history view so at least its possible to analyse the cpu utilization trend. Perfmon has great counters for diagnosing ASP.net application performance issues. There is a very interesting counter to see no of exceptions/sec. Usually support teams tend to simply ignore it. Fixing exceptions is one of those low hanging frutts. Anything over 15 exceptions/sec can start showing great impacts on the performance of your application. High Impact Tools,Managed Stack Explorer – Though some of my peers count it as a low impact tool, but I have had managed stack explorer crash my production application. So, it is not usually in my first few list of tools when diagnosing production issues.Live Debugger – Nothing better than live debugging. But it is important to know that this is absolutely at the extreme end of the impact spectrum. Everytime a live debugger is hit the application execution is interrupted. I would only recommend using this in Pre-Production. Profilers – Profilers are a great tool to narrow down to the root cause of the problem. Performance problems are usually a result of, Poor memory management which could also lead to high CPU utilizationA result of network lag or I/OConcurrency Which is where profilers excel :-)
- Concurrency: Concurrency profiling collects information about multithreaded applications. Resource contention profiling collects detailed call stack information every time that competing threads are forced to wait for access to a shared resource. Concurrency visualization also collects more general information about how your multithreaded application interacts with itself, the hardware, the operating system, and other processes on the host computer:Resource contention reports display the total number of contentions and the total time that was spent waiting for a resource for the modules, functions, source code lines, and instructions in which the waiting occurred. Timeline graphs also show the contentions as they occurred.The concurrency visualizer displays graphical information that you can use to locate performance bottlenecks, CPU underutilization, thread contention, thread migration, synchronization delays, areas of overlapped I/O, and other information. When possible, the graphical output links to call stack and source code data. Concurrency visualization data can be collected only for command line and Windows applications..NET Memory:The .NET memory allocation profiling method interrupts the computer processor at each allocation of a .NET Framework object in a profiled application. When object lifetime data is also collected, the profiler interrupts the processor after each .NET Framework garbage collection.The profiler collects information about the type, size, and number of objects that were created in an allocation or were destroyed in a garbage collection.When an allocation event occurs, the profiler collects additional information about the function call stack. Exclusive allocation counts are incremented for the function that is currently executing and inclusive counts are incremented for all the calling functions on the call stack. .NET reports present the totals of these counts for the profiled types, modules, functions, source code lines, and instructions.When a garbage collection occurs, the profiler collects data about the objects that were destroyed and information about the objects in each garbage collection generation. At the end of the profiling run, the profiler records data about the objects that were not explicitly destroyed. The Object Lifetime report displays the totals for each type that was allocated in the profiling run..NET memory profiling can be used in either sampling or instrumentation mode. The mode that you select does not affect the Allocation and Object Lifetime reports that are unique to.NET memory profiling:When you run .NET memory profiling in sampling mode, the profiler.NET uses memory allocation events as the interval and displays the number of objects that were allocated and the total bytes that were allocated as the inclusive and exclusive values in the reports.When you run .NET memory profiling in instrumentation mode, detailed timing information is collected together with the inclusive and exclusive allocation values.Tier Interaction: Tier-interaction profiling adds information to a profiling data file about synchronous ADO.NET calls between an ASP.NET page or other application and a SQL Server database. Data includes the number and time of calls, and the maximum and minimum times. Tier-interaction data can be added to profiling data that is collected with the sampling, instrumentation, .NET memory, or concurrency methods.
- NOTE - Set the project configuration setting to Release. Do not profile in Debug mode. Talk about the License requirements Breaking changes- Windows Server 2008 and Windows 8 VS the other operating systems, sampling no more supports TIP and additional options
- DemoHow to set up VS standalone profilerA problem tackling approach, start with sampling, analyze sampling results. Narrow down on the problem area via hot path and function count view.Show off Instrumentation to narrow down to the function code, look at the inclusive methods count. Talk about memory management and show off .NET memory management profiling using Instrumentation.Talk about Concurrency visualizer to analyze how multi threaded applications can burn you downDiscuss how as a consultant I benefitted – success story
- Pros and Cons of Visual Studio Standalone profilerhttp://msdn.microsoft.com/en-us/library/bb385759.aspxMicrosoft will be brininging “Java script profiling” capability to the next quarterly update
- And we now reach the most important part of my talk “How to make your boss love you!”
- One of the key reasons why performance Issues are hard to solve is because its very tricky to reproduce these issues in test environments. Performance testing is really a religion, it’s a culture that you can adopt one step at a time. The thumb rule is for any project should be “Start early, start small and target simulating production hardware environment in your test rig”Now talking about the problem at hand, how to easily reproduce production usage of your application in a test environment. What are IIS LogsTo help with server use and analysis, IIS is integrated with several types of log files. These log file formats provide information on a range of websites and specific statistics, including Internet Protocol (IP) addresses, user information and site visits as well as dates, times and queries. If you are using IIS 7 and above you will find the log files in the following directory C:\\Interpub\\Logs\\Log ParserAllow me to introduce you to log parser if you haven’t already heard of it. It is a is a powerful, versatile tool that provides universal query access to text-based data such as log files, XML files and CSV files, as well as key data sources on the Windows operating system such as the Event Log, the Registry, the file system, and Active Directory.More details… http://geekswithblogs.net/TarunArora/archive/2012/07/04/using-iis-logs-for-performance-testing-with-visual-studio.aspx
- Log Parser allows you to query IIS logs. By importing the log parser library in a .net project it is possible to parse the get requests in iis log into url’s. These urls can be programmatically called using the visual studio performance testing API. This is a gold mine! You can now reproduce any production issue in any environment. What more, you can convert this test into a load test and now you can simulate performance issues or simply stress test your application.
- DemoIIS logsLog ParserWalkthrough code base and simulate one test creation> Finish the talk with Load Testing in the Cloud and how to leverage the cloud for performance testing.
- Quick Recap
- Open the stage for any questions. TODO: Add another slide with useful links that users can