Hi! This is Jeff West. Thanks for joining us for the fifth installment in our webcast series for developers. Today we will be covering Troubleshooting and Tuning with WebLogic.
Here is the agenda for the session. First, we will cover troubleshooting and tuning in a general sense and then we will focus on JRockit and WebLogic. After the presentation, we will have a demo of load testing with soapUI, monitoring with WebLogic Console and JRockit Mission Control and then diagnostics with JRockit Flight Recorder.
There are many layers of an application stack that can contribute to poor performance, scalability or throughput. You can take a top-down approach starting with your application and moving down the stack, but the best approach is to take a holistic approach and look at multiple levels of the stack at the same time when you are doing testing and troubleshooting. Many of our customers seem to start at the App Server layer because they claim that nothing is wrong with the applications they build. Most code written by all developers throughout the world has bugs and can be performance tuned, and even if you think you have the best developers in the world, chances are there are opportunities to increase performance and scalability in your application code
The Ideal ‘performance funnel’ is one that is not a funnel – it is a pipe. But this is not reality. At every layer of the stack there is a loss of performance and the task of performance tuning is to minimize the impact that each layer has. Applications that are well designed (and built) will typically bottleneck with I/O, and this is really where you want to be. This means that you have minimized the impact of the other layers and gotten to a physical limit, not a logical one.
In most cases, this is where you will start. You will start with an ‘average’ application running on an app server that has no tuning whatsoever. In this case, chances are the application will be the bottleneck, this is not an accusation, this is just how the process works. First you have to make it work and then you make it work faster. Only a very, very mature team of developers will produce an application that doesn’t require much performance tuning.
There are many steps along the way. Once you clear a bottleneck, you’ll find another one. Sometimes, a bottleneck that you thought you cleared will re-emerge or manifest itself in a different way. It is an iterative process, and while you may allocate 2 weeks for performance testing, there’s no guarantee that you’ll get it all done in that timeframe. As a developer participating in performance testing, this should be your expectation and you shouldn’t lose hope.
When you look at Oracle’s overall strategy, you see an attempt to get the best configuration, best performance, out of the box. We want to control the whole stack – not because we want to control the world – but because we want to be able to perform engineering at each level of the stack in order to provide the best performance and user experience possible. Since we own the whole stack, now with SUN hardware, we can engineer systems that would be very difficult, if not impossible to do if we only had influence over the database, application or middleware layer.
Enough marketing-speak. Lets talk about the tools that come with WebLogic for troubleshooting.
There are many tools available for troubleshooting. You can use Windows performance monitor, Unix TOP and other commands, or packaged software. And as you look at the different layers of the stack you will see that there are many levers to pull and variables to consider. When you are trying to find a problem, you start ruling out factors at each layer of the stack. Sometimes, you can narrow it down, sometimes you can’t. In many cases you find that you have to put in a lot of effort to correlate data from different sources in order to get to a root cause. And then you have to do it over again.
What we will be focusing on today is the set of tools that is available with WebLogic in order to troubleshoot and tune. This set includes the Jrockit Flight Recorder and Jrockit Mission Control.
For those of you who are not familiar with it, I want to give a brief introduction of the JRockit JVM.
JRockit is a JVM that was created by Appeal Virtual Machines which was acquired by BEA in 2002. In 2008 it became part of Oracle Fusion Middleware. It is a proprietary JVM that was written specifically for server-side applications. As such, it is written for machines that have a lot of physical memory and for applications that run for a very long time – days, weeks, even years – such as application servers. If you look at the startup time compared with HotSpot, you will find that in some cases JRockit starts up slower, but JRockit trades startup time and memory for improved performance. JRockit does not support JavaWebStart, but does support Swing applications. Since Desktop Swing applications are short-lived, you may find a better experience using HotSpot for these.
Lets get into the weeds a bit with WebLogic
The WebLogic Diagnostic Framework (WLDF) is a monitoring and diagnostic framework that defines and implements a set of services that run within WebLogic Server® processes and participate in the standard server life cycle. Using WLDF, you can create, collect, analyze, archive, and access diagnostic data generated by a running server and the applications deployed within its containers. This data provides insight into the run-time performance of servers and applications and enables you to isolate and diagnose faults when they occur. WLDF includes several components for collecting and analyzing data: The Diagnostic Image Capture creates a diagnostic snapshot from the server that can be used for post-failure analysis. Archive is used to persists data events, log records, and metrics from server instances and applications over time There are point-cut Instrumentations that add diagnostic code to WebLogic Server instances and the applications running on them to execute diagnostic actions at specified locations in the code. The Instrumentation component provides the means for associating a diagnostic context with requests so they can be tracked as they flow through the system. Mbean Harvester — Captures metrics from run-time MBeans, including WebLogic Server MBeans and custom MBeans, which can be archived and later accessed for viewing historical data. With watches and Notifications we provide the means for monitoring server and application states and sending notifications based on criteria set in the watches. And finally with Logging services you can manage logs for monitoring server, subsystem, and application events. The WebLogic Server logging services are documented separately from the rest of the WebLogic Diagnostic Framework.
I have mentioned the ‘Flight Recorder’ a couple times, and you may be wondering what that is. The JRockit flight recorder is much like the flight recorder in an airplane that records all of the events that happen in the system. The goal is to be able to use this data in the case of a problem or a crash in order to ‘go back in time’ and see what was happening leading up to the problem. It is always on, and is a circular buffer that only retains a certain amount of data. Even though it is always on, it does not have a significant amount of overhead. It also has built-in integration with JRockit Mission Control and replaces the JRA recording and LAT capabilities. It not only includes data from the JVM, but also from the WebLogic Diagnostic framework and the Fusion Middleware Dynamic Monitoring System which allows you to get events for ADF, Web Center and SOA Suite applications,.
Carges_TBAv1_04.01.05.ppt The flight recorder is intended to be non-intrusive when enabled and to provide diagnostic information in PRODUCTION systems. You can also adjust the level of data that is generated for capture higher, but it will have additional overhead.
We have engineered the integration of WebLogic and the Flight Recorder to be able to: View and correlate WebLogic data and events with JVM events and make problem identification easier. We have instrumented WebLogic to generate Flight Recorder events and have added a custom WebLogic event viewed in JRockit Mission Control We also wanted to make it easy to configure and use. It is available out of the box, and it is easy to adjust the volume of events that is generated. We also wanted to be able to leverage the diagnostics in WebLogic server in order to increase the usefulness of the flight recorder so we have added the ability to dump a Flight Recorder dataset as a result of a watch rule being met using the WLDF framework in WebLogic.
The WLDF FlightRecorder events are automatically throttled when the system is under stress. The algorithm that implements this throttling uses two different measurements, and there are "caps" for each of these criteria, currently at 128 req/sec and 800 events/sec, respectively. When these caps are exceeded, requests are sampled at a rate intended to keep things within those thresholds. So basically, when your system is under load, the FR adjusts in order to reduce the overhead.
It is very easy to configure the WLDF event volume, and I’ll show you how we can do that in our demo. There are 4 levels – Off, Low, Medium and High. We consulted with our developer user groups and support organization to come up with what we feel is a good segmentation of the events. In many cases, customers will enable the WLDF at HIGH in development and LOW or MEDIUM in production and they are comfortable with the small amount of overhead.
Here are some of the events that are generated at a ‘low’ volume. Here you will see that events for EJB invocation JDBC statements Servlet Invocation Web Application Load and Unload Web Services JAXRPC request/response JAXWS request/response In many cases this information is sufficient to identify the area(s) for further investigation.
As you go up to medium, you will see events for EJB home create/remove More detailed JDBC connection information And more detailed Servlet information
With HIGH, you will see even more events such as JDBC transaction information JTA transactions More detailed Servlet information And stack traces and user IDs
We have also integrated JFR with WLDF such that you can capture a FR recording as the result of a watch alert using the WLDF. We also provide integration with the Remote Diagnostics Agent such that our support organization can proactively capture a JFR image.
Currently unsupported, but gives you a preview of whats to come. Filtering of WebLogic specific events in the JRMC. We’ll cover this in the demo, so I won’t spend too much time on it now.
Basically, it gives you a WebLogic tab with WebLogic events organized in a user-friendly way. We’ll see this in the demo.