Monitoring VMware vFabric with Hyperic and Spring Insight
•Transferir como PPTX, PDF•
1 gostou•1,045 visualizações
Webinar presented by Steve Millidge, C2B2 Director, on the 19th of November 2012. This webinar demonstrates how you can significantly improve your support of production tcServer environments using Hyperic and Spring Insight. In the webinar C2B2's experienced support engineer will show you how to gather extensive internal metrics from your production tcServer instances, how to configure alerts to warn you of potential problems in your production tcServer cluster and how to diagnose and triage poorly performing transactions with Spring Insight. To watch high quality version of this webinar please visit https://www4.gotomeeting.com/register/941934207
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (19)
Destaque
Destaque (10)
Semelhante a Monitoring VMware vFabric with Hyperic and Spring Insight
Semelhante a Monitoring VMware vFabric with Hyperic and Spring Insight (20)
Mais de C2B2 Consulting
Mais de C2B2 Consulting (20)
Monitoring VMware vFabric with Hyperic and Spring Insight
- 1. Monitoring VMware vFabric with Hyperic and Spring Insight Steve Millidge © C2B2 Consulting Limited 2012 All Rights Reserved
- 2. vFabric tcServer © C2B2 Consulting Limited 2012 All Rights Reserved
- 3. Why vFabric tcServer? • Pre-configured Templates • Many instances from a single binary install • High Performance JDBC connection pool • Improved Diagnostics • JMX management enabled by default • Unix & Windows Service Scripts • Server Management Tools • Latest security patches as binaries • Integration with vFabric components © C2B2 Consulting Limited 2012 All Rights Reserved
- 4. Features over Tomcat • Templates • Single Binary Install • High Performance JDBC Connection Pool • Clustering • Diagnostics • Elastic Memory for Java • Management © C2B2 Consulting Limited 2012 All Rights Reserved
- 5. Diagnostics • Templates • Single Binary Install • High Performance JDBC Connection Pool • Clustering • Diagnostics • Elastic Memory for Java • Management © C2B2 Consulting Limited 2012 All Rights Reserved
- 6. Diagnostics Template ./tcruntime-instance.sh create <instance> -t diagnostics © C2B2 Consulting Limited 2012 All Rights Reserved
- 7. SlowQueryReportJmx • Reports on SLOW and FAILED Queries • Subscribe for JMX Notifications © C2B2 Consulting Limited 2012 All Rights Reserved
- 8. ThreadDiagnosticsValve $CATALINA_BASE/logs/catalina.YYYY-MM-DD.log © C2B2 Consulting Limited 2012 All Rights Reserved
- 9. + Extended Logging © C2B2 Consulting Limited 2012 All Rights Reserved
- 10. JMX Notifications • Reports on “slow”, “stuck” and “failed” requests • Subscribe for JMX notifications or view log files © C2B2 Consulting Limited 2012 All Rights Reserved
- 11. EM4J • Templates • Single Binary Install • High Performance JDBC Connection Pool • Clustering • Diagnostics • Elastic Memory for Java • Management © C2B2 Consulting Limited 2012 All Rights Reserved
- 12. Traditional Best Practice for Virtualised JVM’s • Reserve 100% of the required VM memory • Tune JVM Heap to occupy minimum • Avoid GC Thrashing “Both ballooning and swapping should be prevented for Java application” • No benefit from Virtualisation memory sharing • All resources must be provisioned upfront © C2B2 Consulting Limited 2012 All Rights Reserved
- 13. ESXi 5.0 Memory Reclamation • Transparent Page Sharing • Memory Compression • VMware Tools Balloon OR EM4J Balloon • Swap © C2B2 Consulting Limited 2012 All Rights Reserved
- 14. EM4J Balloon Driver Hypervisor Guest OS Guest OS JVM Heap JVM Heap JVM Heap JVM Heap ... © C2B2 Consulting Limited 2012 All Rights Reserved
- 15. Elastic Memory for Java EM4J • Requires ESXi 5.0 • Redhat / Centos v5 (x32 or x64) • Hotspot JDK 1.6 • Heap > 512Mb • Recommend no more than 4 JVMs per VM • JVMs should account for the majority of the VM memory usage © C2B2 Consulting Limited 2012 All Rights Reserved
- 16. Switch EM4J On • Enable EM4J in the VM – sched.mem.pshare.guestHintsSyncEnable=true – sched.mem.maxmemctl=-1 (max balloon size) • Create EM4J instance – ./tcruntime-instance.sh create instanceName -t elastic- memory © C2B2 Consulting Limited 2012 All Rights Reserved
- 17. Troubleshooting your App • How many database calls does that page generate? • What was the SQL executed? • How long did the query take? • Why does this page take a long time to load? • Which part was slow? • What happened then?! © C2B2 Consulting Limited 2012 All Rights Reserved
- 18. Spring Insight to the Rescue What does it provide? • Information on all database queries • Information on all web request and response information • Lists component calls and parameters Where can I use it? • Agile Development • QA • Load Testing • Production Monitoring © C2B2 Consulting Limited 2012 All Rights Reserved
- 19. Under the Covers How does it work? • Agent instruments application code using AspectJ • Reports to “dashboard” (insight.war) component responsible for rendering statistics Increased Memory Requirements • Increased memory –Xmx • Increased permgen –XX:MaxPermSize © C2B2 Consulting Limited 2012 All Rights Reserved
- 20. Applications Summary © C2B2 Consulting Limited 2012 All Rights Reserved
- 21. Application View © C2B2 Consulting Limited 2012 All Rights Reserved
- 22. End Point View © C2B2 Consulting Limited 2012 All Rights Reserved
- 23. Response Time Histogram © C2B2 Consulting Limited 2012 All Rights Reserved
- 24. Request & Response Summary © C2B2 Consulting Limited 2012 All Rights Reserved
- 25. JDBC Query © C2B2 Consulting Limited 2012 All Rights Reserved
- 26. Recent Activity © C2B2 Consulting Limited 2012 All Rights Reserved
- 27. vFabric Hyperic Integration More later! © C2B2 Consulting Limited 2012 All Rights Reserved
- 28. vFabric Hyperic © C2B2 Consulting Limited 2012 All Rights Reserved
- 29. Overview • Monitoring & Management Solution • Out-of-box monitors over 75 technologies • Automatic discovery of resources • Metric Collection • Event Tracking • Resource Control • Alerting & Notifications • Access to Live Information © C2B2 Consulting Limited 2012 All Rights Reserved
- 30. Architecture © C2B2 Consulting Limited 2012 All Rights Reserved
- 31. Resource Hierarchy • Platform – Win2008, linux etc • Platform Services – CPU – File System Mount – Network Interfaces • Server – tcServer instance – RabbitMQ Server • Service – tcServer Session Manager – RabbitMQ Queue © C2B2 Consulting Limited 2012 All Rights Reserved
- 32. Hyperic User Interface © C2B2 Consulting Limited 2012 All Rights Reserved
- 33. Resource View © C2B2 Consulting Limited 2012 All Rights Reserved
- 34. Resource Alerts • Metric Alerting • Event Alerting – Log file tracking – Config file tracking © C2B2 Consulting Limited 2012 All Rights Reserved
- 35. tcServer Application Management © C2B2 Consulting Limited 2012 All Rights Reserved
- 36. vSphere Monitoring • Server side HQU Extension • Remote agent accesses vSphere vCenter © C2B2 Consulting Limited 2012 All Rights Reserved
- 37. vCenter Resource Hierarchy Agent monitoring vCenter Agent installed on VM • vCenter – Software • Platform • ESX Host – Platform – Win2008, linux etc • VM – Platform • Platform Services – CPU Features – File System Mount • Auto disovery of ESX – Network Interfaces Hosts and VM’s • Server • Auto detection of new – tcServer instance ESX Hosts and VM’s – RabbitMQ Server • Auto detection of • Service architecture changes e.g. – tcServer Session Manager vMotion – RabbitMQ Queue © C2B2 Consulting Limited 2012 All Rights Reserved
- 38. ESX Host & VM Metics © C2B2 Consulting Limited 2012 All Rights Reserved
- 39. Quest © C2B2 Consulting Limited 2012 All Rights Reserved
Notas do Editor
- Hi and welcome to this short presentation regarding our Middleware Expert Support offering.
- Compared to basic TomactAlso get Support from VMWare – who have many commiters. We can then provide deep operational support to your system running on tcServer.
- Go through these individually
- NOTE: that the diagnostics template is WRONGTo enable JMX notifications for the SlowQueryReport YOU MUST DECLARE THE factory as:<!--========================================================================================================================================NOTE: to enable the SlowQueryReportJmx JMX notifications YOU MUST USE the standard org.apache.tomcat.jdbc.pool.DataSourceFactory factory======================================================================================================================================== factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" factory="com.springsource.tcserver.serviceability.request.DataSourceFactory"its got something to do with the MBean getting bound under "tomcat.jdbc:" namespace rather than the "tcserver:" namespace-->
- Allows viewing of request headersTraces for “stuck” queriesThread information
- Historically if the Hypervisor swapped guest OS memory or re-distributed by use of the guest OS balloon driver then this could directly impact on JVM GC performance. Protracted GC full collections while the JVM attempts to mark objects in memory that has been swapped or redistributed.Taken from http://www.vmware.com/files/pdf/techpaper/Enterprise-Java-Applications-on-VMware-Best-Practices-Guide.pdf The -X m x value is the value that you found during load testing for your application on physical servers. This value does not need to change when moving to a virtualized environment. Load testing your application when deployed on vSphere will help confirm the best -X m x• It is recommended that you do not overcommit memory because the JVM memory is an active space where objects are constantly being created and garbage collected. Such an active memory space requires its memory to be available all the time. If you overcommit memory ballooning or swapping may occur and impede performance. value.• ESX host employs two distinct techniques for dynamically expanding or contracting the amount of memory allocated to virtual machines. The first method is known as memory balloon driver (vmmemctl). This is loaded from the VMware Tools package into the guest operating system running in a virtual machine. The second method involvespaging from a virtual machine to a server swap file, without any involvement by the guest operating system.In the page swapping method, when you power on a virtual machine, a corresponding swap file is created and placed in the same location as the virtual machine configuration file (V MX file). The virtual machine can power on only when the swap file is available. ESX hosts use swapping to forcibly reclaim memory from a virtual machine when no balloon driver is available. The balloon driver may be unavailable either because VMware Tools is not installed, or because the driver isdisabled or not running. For optimum performance, ESX uses the balloon approach whenever possible. However, swapping is used when the driver is temporarily unable to reclaim memory quickly enough to satisfy current system demands. Because the memory is being swapped out to disk, there is a significant performance penalty when the swapping technique is used. Therefore, it is recommended that the balloon driver is always enabled, but monitor to verify that it is not getting invoked as that memory is overcommitted.Both ballooning and swapping should be prevented for Java applications. To prevent ballooning and swapping, refer to BP5 – Set memory reservation for VM needs
- Transparent Page SharingESXi constantly scans memory pages in virtual machines for duplicate contents and can reduce memory overhead by eliminating the duplication. The indirection between virtual memory pages in the virtual machines and physical memory pages on the host make this possible. Duplicate pages are mapped to the same physical page, and if a virtual page is modified, a unique new copy is created transparently. For example, initialized memory pages can be shared if they are written with zeros and static code in virtual machines running the same OS and applications. When a virtual machine writes to a shared page, it is given a unique copy, which breaks the sharing for that page. The pages reclaimed by transparent page sharing can significantly improve your consolidation ratio.Memory CompressionESXi employs memory compression to avoid swapping when memory becomes tight. To free memory, ESXi attempts to compress memory pages and save them in a compression cache. Compressing and decompressing uses CPU cycles, but is still far more efficient than disk I/O. Memory compression is the last opportunity to reclaim memory before disk swapping.VMware Tools Balloon (vmmemctl)There is an important separation of concerns between ESXi and the virtual machines running on it; the memory management techniques employed within a virtual machine are completely opaque to the hypervisor. The hypervisor therefore has no knowledge of how memory allocated to a virtual machine is being used and when it swaps memory to disk or compresses memory, it could swap out memory that is still in use. A far more efficient mechanism of reclaiming memory from a virtual machine is the VMware Tools balloon driver, which runs as a process within the virtual machine, allocates and pins unused memory, and communicates the pages back to the hypervisor. The memory owned by the balloon driver can then be temporarily de-coupled from the virtual machine and used elsewhere.ESXi gives the balloon driver a target size, and the balloon driver attempts to fulfill the request. Under the control of the ESXi hypervisor, balloons in each host virtual machine expand or shrink depending on the shifting requirements of the virtual machines. ESXi calculates balloon targets based on virtual machine activity; a less active virtual machine gets a higher balloon target and the reclaimed memory moves to the more active virtual machines.A balloon in the guest operating system forces the OS to use memory more conservatively. Operating systems use every bit of available memory to improve system performance. However, caching inactive data in one virtual machine while another virtual machine has scarce memory for immediate needs is poor use of memory in a virtualization environment. Expanding the balloon in a virtual machine constrains the OS from using memory for low priority storage and allows the hypervisor to distribute physical memory resources where they are most needed.EM4J BalloonWhen a Java application starts, the operating system allocates memory to the JVM. The allocated memory is then managed by the JVM. On a virtual machine deployed to serve an enterprise Java application, the Java heap typically occupies the greatest portion of allocated memory. The separation of concerns between the operating system and the JVM is very similar to that between the ESXi hypervisor and virtual machines. The JVM manages its object heap as a single block of memory which is entirely opaque to the OS. When Java objects become garbage, there is no way that the OS can reclaim this memory, and if the OS cannot reclaim the memory, then neither can the VMware tools balloon. For this reason, in order for the hypervisor to reclaim memory from a virtual machine in which the Java heap occupies the majority of the available memory, a balloon operating within the Java heap can be much more efficient.When EM4J is enabled in the virtual machine and a JVM executing in the virtual machine, it serves ESXi hypervisor ballooning requests from the JVM heap instead of from the OS memory pool. Just as the guest balloon allocates memory from the OS memory pool, EM4J allocates memory from the Java heap by creating and storing a few large Java objects. To avoid fragmentation, EM4J does not pin memory in the Java heap. The ESXi hypervisor sets the target size of the balloon, and EM4J attempts to satisfy the request.Like the VMware tools guest balloon, the EM4J balloon encourages the JVM to use the available memory more conservatively. When the EM4J balloon inflates, it can force the JVM to clean up garbage and return memory to the hypervisor, typically during periods of low activity. When the JVM needs the memory, typically during an increase in activity, the next garbage collection clears out some of the balloon and that memory is once again available to the JVM. This is the "elastic" nature of EM4J memory management; the JVMs that really need the memory take it from JVMs that would otherwise be wasting it.A welcome side effect of EM4J is that you can size the Java heap less conservatively without wasting memory. Traditionally, to determine the optimum size of the Java heap, you determined the application's peak requirement. Allocating less could mean unacceptable performance, errors, or crashes; allocating more potentially wastes memory. With EM4J you can size the heap to comfortably accommodate the peak requirement, and excess memory is ballooned away and redistributed where it is needed.
- EM4J Considerations and LimitationsWhen you enable EM4J in a virtual machine, memory outside of the Java heap cannot be reclaimed. Therefore, EM4J should only be used in configurations where the majority of memory is used by Java heap.EM4J does not function in heap sizes smaller than 512MB.EM4J currently can reclaim a maximum of 2GB from a Java heap.EM4J works equally well in virtual machines with large pages enabled as with small pages.You can run more than one EM4J-enabled tc Runtime instance in a virtual machine. Each instance contributes to the overall balloon for the virtual machine. However, the individual instances do not coordinate ballooning with each other, so running large numbers of instances is not currently recommended. A pragmatic limit is four EM4J-enabled tc Runtime instances.
- Configure an EM4J-Enabled tc Runtime InstanceThe tc Server elastic-memory template installs a Java library (balloon.jar) and a native library (libballoon.so). It sets up logging for EM4J and modifies the Java command line in the following ways:Sets maximum heap size to 1GB (-Xmx1024M)Sets the java.agent system property to CATALINA_BASE/lib/balloon.jarSets the agent.path.unix.x86 system property to CATALINA_BASE/bin/x86-linux/libballoon.soSets the agent.path.unix.x64 system property to CATALINA_BASE/bin/amd64-linux/libballoon.soTo modify these values, edit the CATALINA_BASE/bin/setenv.sh file, which constructs the JAVA_OPTS environment variable passed to the Java command line when the tc Runtime instance starts up.The elastic-memory template sets the logging level for EM4J to the INFO level by default. You can edit CATALINA_BASE/conf/login.properties to set a different logging level.
- Agent installs onPhysical hostVirtual hostWhere installation is not possible or may not be desirable for performance reasons the agent can be configured to collect metrics across a networkFor exampleNetwork storage devicesNetwork switchesvSpherevCenter
- Change metric display rangeView – customise which charts and what order they are displayed in