Sherlock Homepage - A detective story about running large web services (VISUG 10 years)

Developer Advocate em JetBrains
3 de Feb de 2016
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
Sherlock Homepage - A detective story about running large web services (VISUG 10 years)
1 de 58

Mais conteúdo relacionado

Mais procurados

Tips and Tricks For Faster Asp.NET and MVC ApplicationsTips and Tricks For Faster Asp.NET and MVC Applications
Tips and Tricks For Faster Asp.NET and MVC ApplicationsSarvesh Kushwaha
Batching and Java EE (jdk.io)Batching and Java EE (jdk.io)
Batching and Java EE (jdk.io)Ryan Cuprak
Multi Client Development with SpringMulti Client Development with Spring
Multi Client Development with SpringJoshua Long
Java script frameworkJava script framework
Java script frameworkDebajani Mohanty
Simple REST with DropwizardSimple REST with Dropwizard
Simple REST with DropwizardAndrei Savu
the Spring 4 updatethe Spring 4 update
the Spring 4 updateJoshua Long

Similar a Sherlock Homepage - A detective story about running large web services (VISUG 10 years)

Sherlock Homepage - A detective story about running large web services - WebN...Sherlock Homepage - A detective story about running large web services - WebN...
Sherlock Homepage - A detective story about running large web services - WebN...Maarten Balliauw
Building Continuous Application with Structured Streaming and Real-Time Data ...Building Continuous Application with Structured Streaming and Real-Time Data ...
Building Continuous Application with Structured Streaming and Real-Time Data ...Databricks
Scaling asp.net websites to millions of usersScaling asp.net websites to millions of users
Scaling asp.net websites to millions of usersoazabir
How and why we evolved a legacy Java web application to Scala... and we are s...How and why we evolved a legacy Java web application to Scala... and we are s...
How and why we evolved a legacy Java web application to Scala... and we are s...Katia Aresti
Express node jsExpress node js
Express node jsYashprit Singh
(WEB301) Operational Web Log Analysis | AWS re:Invent 2014(WEB301) Operational Web Log Analysis | AWS re:Invent 2014
(WEB301) Operational Web Log Analysis | AWS re:Invent 2014Amazon Web Services

Similar a Sherlock Homepage - A detective story about running large web services (VISUG 10 years)(20)

Mais de Maarten Balliauw

Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos s...Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos s...
Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos s...Maarten Balliauw
Building a friendly .NET SDK to connect to SpaceBuilding a friendly .NET SDK to connect to Space
Building a friendly .NET SDK to connect to SpaceMaarten Balliauw
Microservices for building an IDE - The innards of JetBrains Rider - NDC Oslo...Microservices for building an IDE - The innards of JetBrains Rider - NDC Oslo...
Microservices for building an IDE - The innards of JetBrains Rider - NDC Oslo...Maarten Balliauw
Indexing and searching NuGet.org with Azure Functions and Search - .NET fwday...Indexing and searching NuGet.org with Azure Functions and Search - .NET fwday...
Indexing and searching NuGet.org with Azure Functions and Search - .NET fwday...Maarten Balliauw
NDC Sydney 2019 - Microservices for building an IDE – The innards of JetBrain...NDC Sydney 2019 - Microservices for building an IDE – The innards of JetBrain...
NDC Sydney 2019 - Microservices for building an IDE – The innards of JetBrain...Maarten Balliauw
JetBrains Australia 2019 - Exploring .NET’s memory management – a trip down m...JetBrains Australia 2019 - Exploring .NET’s memory management – a trip down m...
JetBrains Australia 2019 - Exploring .NET’s memory management – a trip down m...Maarten Balliauw

Mais de Maarten Balliauw(20)

Último

Die ultimative Anleitung für HCL Nomad Web AdministratorenDie ultimative Anleitung für HCL Nomad Web Administratoren
Die ultimative Anleitung für HCL Nomad Web Administratorenpanagenda
Nymity Framework: Privacy & Data Protection Update in 7 StatesNymity Framework: Privacy & Data Protection Update in 7 States
Nymity Framework: Privacy & Data Protection Update in 7 StatesTrustArc
Machine learning with quantum computersMachine learning with quantum computers
Machine learning with quantum computersSpeck&Tech
Product Research Presentation-Maidy Veloso.pptxProduct Research Presentation-Maidy Veloso.pptx
Product Research Presentation-Maidy Veloso.pptxMaidyVeloso
Connecting Africa.docxConnecting Africa.docx
Connecting Africa.docxEric Annan
How to use the Cataloguing Code Ethics at your day job : a hands-on workshop ...How to use the Cataloguing Code Ethics at your day job : a hands-on workshop ...
How to use the Cataloguing Code Ethics at your day job : a hands-on workshop ...CILIP MDG

Último(20)

Sherlock Homepage - A detective story about running large web services (VISUG 10 years)

Notas do Editor

  1. One of these services was failing – resulting in status 500 – resulting in LB removing the instance from pool – all were removed from pool…
  2. Serial killers are like bugs in the societal machine. They hide. They blend in. They can pass for "normal" which makes them tough to find. They attack weakness causing untold damage until caught. And they will keep causing damage until caught. They are always hunting for opportunity.
  3. Yes it’s fun to work on new things. But try figuring out this stuff. Builds experience.
  4. Profiling the search service issue - search service issue * Open capture1.dtt * Notice lots of lock contention * Disable system methods - notice SearcherManager.Get() is top suspect * Unfortunately, this code was deployed without symbols and we no longer had the sources around * Decided to decompile using dotPeek and walk through code - which revealed something (next slide)
  5. Profiling the search service issue - search service issue * Open capture1.dtt * Notice lots of lock contention * Disable system methods - notice SearcherManager.Get() is top suspect * Unfortunately, this code was deployed without symbols and we no longer had the sources around * Decided to decompile using dotPeek and walk through code - which revealed something (next slide)
  6. Perhaps do a quick tour of AppInsights depending on time left.
  7. Analyzing a crash dump * Open w3wp.exe.3344.dmp - explain this can be done with hardcore debugging tools but VS2015 works just as fine * Explain dump summary - what we can see (threads, modules, last state when it all broke down) * Becomes more interesting when we debug it - "Debug with managed only" * Unhandled Exception, it says! That's probably the cause for IIS erroring out. But where does it come from... * We can try the various windows for inspecting modules, threads, call stacks, ... but not al ot to see in there. * We need DEBUGGER SYMBOLS! Unfortunately in this case the gallery was deployed without symbols. No symbols on teh build server either, for this deployment. * DotPeek! Load the actual DLL, enable symbol server, start debugging again. * See the exception now shows us the (decompiled) code. Not the real code but it does give us an idea. * Unhandled exception in a task. Which crashes out IIS.
  8. Reproducing production traffic * We'll use jMeter - a tool that is perfect for replaying web server logs against other servers * Explain IIS logs come from production * Explain rconvlog tool to translate them to NCSA format * Open up jMeter and explain how it is all linked to each other
  9. Application roots: Typically, these are global and static object pointers, local variables, and CPU registers.
  10. Application roots: Typically, these are global and static object pointers, local variables, and CPU registers.
  11. Application roots: Typically, these are global and static object pointers, local variables, and CPU registers.
  12. Application roots: Typically, these are global and static object pointers, local variables, and CPU registers.
  13. Application roots: Typically, these are global and static object pointers, local variables, and CPU registers.
  14. Analyzing memory usage * Open the Workspace [2015-08-05] [13-24].dmw workspace * From the snapshots overview we already see a few interesting things: memory keeps going up, when GC'ed (using the profiler API) memory stays around * Let's open one of the snapshots - snapshot #6 - why? As it says objects have been collected so we want to see what remained in memory * From the overview: lots of objects on gen2 - eaning they survived many collections - dive in * Seems NInject.Activation.Caching.Cache is keeping a lot of bytes in memory - dive in * Lots of objects seem to be cached by NInjet. Check outgoing references to see what they are. * The first ones look normal, e.g. the controller activator and so on is needed the entire time by ASP.NET MVC * At [2] we see an EntitiesContext retained. That's weird: it should be disposed of after each request. Let's see if there are more of these still in memory. * Snapshot - largest size - search entitiescontext - A LOT! Also in size relative to the snapshot size - dive in * We can see 991 instances are kept around - who's holding on to them? * Group by similar retention shows NInject for 596 of them. Wow.
  15. Yes it’s fun to work on new things. But try