Monitoring and analytics with was liberty

© 2015 IBM Corporation
Monitoring and Analytics with
the WebSphere Liberty
Profile
Don Bourne

Agenda
• Runtime Self-Checks
− Slow / Hung Request Detection
− Event Logs
• Performance Monitoring
– Monitor MXBeans
– Viewing Data in Jconsole
– Viewing Data in Admin Center
• Embedded Analytics
− Access Logs
− Message Logs / Binary Logs / FFDC
2

Request Timing
 Which requests are slow?
 Why are they slow?
4
NEW IN FEBRUARY 2015 LIBERTY BETA

5
Request Timing – Concepts
 Built on a new request probe framework
– Tracks REQUESTS made to the
application server
• Requests have a unique request
ID
• Requests have a tree of events
– Builds a tree of EVENTS as each
request runs
• Events have a type that identifies
what kind of event it is
• Events have context info that
describes an instance of the event
• Events have a duration
Servlet service
EJB method
JDBC query
JDBC update
EJB method
A single request with a tree of events
RequestID: AAC9KLwFFXT_AAAAAAAAAAN

6
Request Timing – What does it do?
Slow requests
 Logs the request with the tree of events
 Logs the current stack for the thread the request is using
 Repeats 3 times
Hanging requests
 Logs the request with the tree of events
 Captures a series of 3 javacores
Noteworthy
 Does not attempt to abort requests
 Reports on work from a single thread
 Production-friendly
 Idle-friendly

7
Request Timing – Configuration
<featureManager>
<feature>requestTiming-1.0</feature>
</featureManager>
<requestTiming
slowRequestThreshold="10s"
hungRequestThreshold="600s"
includeContextInfo="true"
sampleRate="1"
<timing
type="websphere.servlet.execute"
slowRequestThreshold="20s"
hungRequestThreshold="900s"
/>
/>

9
Request Timing - Output
Sample slow request detection output:
TRAS0112W: Request websphere.servlet.service|DayTrader Web | TradeScenarioServlet(AAC9KLwFFXT_AAAAAAAAAAN) has been
running on thread 0000006b for 1549.460ms. The following stack trace shows what this thread is currently doing.
<stack trace>
The following table shows the events that have run during this request.
Duration Operation
1552.012ms + websphere.servlet.service | DayTrader Web | TradeScenarioServlet
0.014ms websphere.session.getAttribute | R-ObCtcDfR8Zd9riQEMCh6R | uidBean
30.714ms websphere.servlet.service | DayTrader Web | TradeAppServlet
30.456ms websphere.servlet.service | DayTrader Web | /quote.jsp
28.903ms websphere.servlet.service | DayTrader Web | /displayQuote.jsp
0.194ms websphere.datasource.psExecuteQuery | jdbc/TradeDataSource | SELECT t0.CHANGE1, t0.COMPANYNAME, t0.H
1520.695ms + websphere.servlet.service | DayTrader Web | TradeAppServlet
0.190ms websphere.datasource.psExecuteQuery | jdbc/TradeDataSource | SELECT t0.ADDRESS, t0.CREDITCARD, t0.EMAIL, t0
0.135ms websphere.datasource.psExecuteQuery | jdbc/TradeDataSource | SELECT t0.ACCOUNTID, t0.BALANCE, t0.CREATIONDAT
0.110ms websphere.datasource.psExecuteQuery | jdbc/TradeDataSource | SELECT t0.CHANGE1, t0.COMPANYNAME, t0.HIGH, t0
0.299ms websphere.datasource.psExecuteQuery | jdbc/NoTxTradeDataSource | SELECT KEYVAL FROM KEYGENEJB WHERE KEYNAME
0.571ms websphere.datasource.psExecuteUpdate | jdbc/NoTxTradeDataSource | UPDATE KEYGENEJB SET KEYVAL = ? WHERE KEYN
0.126ms websphere.datasource.psExecuteQuery | jdbc/NoTxTradeDataSource | SELECT KEYVAL FROM KEYGENEJB WHERE KEYNAME
0.286ms websphere.datasource.psExecuteUpdate | jdbc/NoTxTradeDataSource | UPDATE KEYGENEJB SET KEYVAL = ? WHERE KEYN
522.450ms websphere.datasource.psExecuteUpdate | jdbc/TradeDataSource | INSERT INTO orderejb (ORDERID, COMPLETIONDATE,
230.333ms websphere.datasource.psExecuteUpdate | jdbc/TradeDataSource | INSERT INTO holdingejb (HOLDINGID, PURCHASEDAT
0.936ms websphere.datasource.psExecuteUpdate | jdbc/TradeDataSource | UPDATE accountejb SET BALANCE = ? WHERE ACCOUN

10
Request Timing - Output
Sample hung request detection output:
TRAS0114W: Request AAAg7irff7c_AAAAAAAAAAA has been running on thread 000000c9 for 600579.325ms. The following table shows
the events that have run during this request.
<event tree>
CWWKE0067I: Java dump request received.
JVMDUMP034I User requested Java dump using 'd:wlpusrserverss1javacore.20150220.155505.13168.0001.txt' through
com.ibm.jvm.Dump.JavaDump
JVMDUMP010I Java dump written to d:wlpusrserverss1javacore.20150220.155505.13168.0001.txt
CWWKE0068I: Java dump created: d:wlpusrserverss1javacore.20150220.155505.13168.0001.txt
<1 minute passes>
<1 minute passes>
TRAS0115W: Request AAAg7irff7c_AAAAAAAAAAA, which was previously detected to be hung, has completed after 821583.742ms.

Event Logging
 Logs data at start and/or end of events
− Event type / Event context info1
− Event duration1
− Request ID1
 Can be configured to
− Log at entry, exit, or both
− Log only for specified list of event types
− Log only for events longer than a certain duration (for logs written at event
exit only)
 Log events will use log record extensions (filterable using binaryLog
command) for
− Type
− Context info
− Request ID
11
1Request and event details are provided by underlying Liberty Request Probe framework

Event Logging – Log Record Context
Key event logging fields are added to the LogRecordContext
 requestID
 type
 contextInfo
Fields from LogRecordContext are stored in log / trace records
when binary logging is used
The wlp/bin/binaryLog command can filter for records by extension:
binaryLog view myServer –-includeExtension="type=websphere.datasource.executeUpdate"
binaryLog view myServer –-includeExtension="type=websphere.datasource.executeUpdate"
--format=advanced
12

Event Logging – Configuration
13
<featureManager>
<feature>eventLogging-1.0</feature>
</featureManager>
<eventLogging
includeTypes="websphere.servlet.execute"
minDuration="500ms"
logMode="exit"
sampleRate="1"
includeContextInfo="true"
/>

Event Logging
Sample event logging output (basic format):
Servlet:
[6/18/14 16:21:35:761 IST] 0000002a EventLoggingProbeExtension.class I BEGIN
requestID=AAADvUHkFwy-AAAAAAAAAAD # type=websphere.servlet.service # contextInfo=com.ibm.ws.request.timing.TestJDBC
[6/18/14 16:22:04:643 IST] 0000002a EventLoggingProbeExtension.class I END
requestID=AAADvUHkFwy-AAAAAAAAAAD # type=websphere.servlet.service # contextInfo=com.ibm.ws.request.timing.TestJDBC #
duration=2.614ms
JDBC:
requestID=AAADvUHkFwy-AAAAAAAAAAD # type=websphere.datasource.executeUpdate # contextInfo=jdbc/exampleDS | create table
cities (name varchar(50) not null, population int, county varchar(30))
requestID=AAADvUHkFwy-AAAAAAAAAAD # type=websphere.datasource.executeUpdate # contextInfo=jdbc/exampleDS | create table
cities (name varchar(50) not null, population int, county varchar(30)) # duration=0.231ms
Session:
requestID=AAADvUHkFwy-AAAAAAAAAAE # type=websphere.session.sessionReleased # contextInfo=default_host/jdbcTestPrj_3
requestID=AAADvUHkFwy-AAAAAAAAAAE # type=websphere.session.sessionReleased # contextInfo=default_host/jdbcTestPrj_3 #
duration=0.012ms
[note that the contextInfo for servlets includes servlet name, pathinfo and query string when that info is present in
the request]
15

Event Logging
Zooming in...(JDBC example)
requestID=AAADvUHkFwy-AAAAAAAAAAD
type=websphere.datasource.executeUpdate
contextInfo=jdbc/exampleDS | create table
cities (name varchar(50) not null,
population int, county varchar(30))
duration=0.231ms
16
[6/18/14 16:21:44:200 IST] 0000002a EventLoggingProbeExtension.class I END requestID=AAADvUHkFwy-
AAAAAAAAAAD # type=websphere.datasource.executeUpdate # contextInfo=jdbc/exampleDS | create table cities (name varchar(50) not
null, population int, county varchar(30)) # duration=0.231ms

Performance Monitoring
Focus Areas
 JVM Performance : Analysis on JVM Performance Data , GC data
 Application Performance : Servlet Usage and response time, Sessions
 WAS Containers : Usages of ThreadPools, ConnectionPools
monitor-1.0 feature (introduced in WAS85 Liberty Profile)
 Collects data from various Liberty components
 Data reporting using MXBeans
 MXBean data can be viewed with JConsole1 or any standard JDK client
 Much of the data can also be viewed from Liberty admin center tools
18

Performance Monitoring
JConsole (part of JDK)
 Graphical monitoring tool to monitor Java Virtual Machine (JVM) and java applications
 Comes as part of Java Development Kit (java/bin/jconsole)
 Connects to local or remote JVMs
 Interacts with JMX MBeans and MXBeans to get data or execute operations
 Provides a way to log counters into a CSV file
19

monitor-1.0
 The monitor-1.0 feature monitors the following components:
− JVM JvmStats MXBean
− Web applications ServletStats MXBean
− Thread pool ThreadPoolStats MXBean
− [8.5.5.0 or later] JAX-WS endpoints WebServiceStats MXBean
− [8.5.5.0 or later] Session management SessionStats MXBean
− [8.5.5.0 or later] ConnectionPool ConnectionPool MXBean
 All MXBeans in a collective can be accessed by connecting to the collective
controller using the JMX REST Connector and the RoutingContext MBean1
1see http://www-01.ibm.com/support/knowledgecenter/SSD28V_8.5.5/com.ibm.websphere.wlp.nd.doc/ae/rwlp_jmx_routing.html
20

monitor-1.0
JvmStats
 Sample Object Name: WebSphere:type=JvmStats
− Heap
− FreeMemory
− UsedMemory
21
− ProcessCPU
− GcCount
− GcTime
− UpTime

monitor-1.0
ServletStats (1 MXBean per servlet)
 Sample Object Name: WebSphere:type=ServletStats,name=MyApp.MyServlet
− AppName
− ServletName
− RequestCount
− ResponseTime (nanoseconds)
22
− Description
− RequestCountDetails
− ResponseTimeDetails

monitor-1.0
ThreadPoolStats
 Sample Object Name: WebSphere:type=ThreadPoolStats,name=Default Executor
− ActiveThreads
− PoolSize
− PoolName
23

monitor-1.0
SessionStats (1 MXBean per web application)
 Sample Object Name: WebSphere:type=SessionStats,name=default_host/trade_lite
− CreateCount
− LiveCount
− ActiveCount
24
− InvalidatedCount
− InvalidatedCountbyTimeout

monitor-1.0
ConnectionPool (1 MXBean per connection pool)
 Sample Object Name: WebSphere:type=ConnectionPool,name=jdbc/TradeDataSource
− CreateCount
− DestroyCount
− ManagedConnectionCount
25
− WaitTime
− ConnectionHandleCount
− FreeConnectionCount

Monitoring Views in the Admin Center Explore
Tool
26

monitor-1.0
Traditional PMI Support
(for clients written to work with the full profile application server)
− monitoring data is available in PMI format (CountStatistics,
RangeStatistics, etc.) via “Perf” Mbean “WebSphere:*,type=perf”
− Existing clients will be able to work (changes required in
connectors)
 Use JMX Rest Connectors to connect to Liberty
 Retrieve all MBeans
 Query for MBean type=Perf
 Invoke standard methods exposed by “Perf” MBean
27

monitor-1.0 - Configuration
28
<featureManager>
<feature>monitor-1.0</feature>
</featureManager>
<monitor
// Property to enable or disable Traditional PMI way of reporting.
enableTraditionalPMI="false"
// Allows user to enable/disable monitors based on group name such as
// WebContainer,JVM,ThreadPool,Session,ConnectionPool,...
filter="Session"
/>

monitor-1.0
Performance
 Overhead is about 4%, when all components are monitored
 Overhead is lower when fewer components are monitored
29

Liberty Integrated Log
Analytics

31
Liberty Integrated Log Analytics
 Data and analytics based on content
from logs / trace / ffdc / access logs
Liberty (user apps)
AnalyticsCollector-1.0
AdminCenter
AnalyticsEngine-1.0
messages.log
trace.log
access.log
binary log
ffdc files
Index
kibana.war*
Admin Center Explore
Tool
Kibana (not included with Liberty)

32
Integrated Log Analytics – Configuration
<featureManager>
<feature>analyticsCollector-1.0</feature>
<feature>analyticsEngine-1.0</feature>
</featureManager>
<analyticsCollector
logLevel="INFO"
accessLogEnabled="true"
ffdcEnabled="true"
/>
<httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="9085"
httpsPort="9448">
<accessLogging
filepath="${server.output.dir}/logs/http_defaultEndpoint_access.log"
logFormat="%h %u %t "%r" %s %b %D %{User-agent}i"
/>
</httpEndpoint>

34
Integrated Log Analytics – Admin Center Views
Admin Center Explore Tool contains analytic views and monitoring data

35
Integrated Log Analytics – Sample Kibana Dashboard
We've experimented with deploying Kibana in the same server as our analytics engine
(Kibana is not included with Liberty)

Useful Links – Full Profile
36
Using High Performance Extensible Logging to troubleshoot applications
http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp?topic=%2Fcom.ibm.websphere.nd.multiplatform.doc%2Fae%2Fttrb_usinghpel.html
High Performance Extensible Logging (HPEL)
http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/index.jsp?topic=/com.ibm.iea.was_v8/was/8.0/ProblemDetermination/WASv8_HPEL/player.html
System administration in WebSphere Application Server V8.5, Part 3
High Performance Extensible Logging (HPEL): http://www.ibm.com/developerworks/websphere/techjournal/1208_bourne/1208_bourne.html
WebSphere Application Server v8.5 Cross Component Trace
http://webspherecommunity.blogspot.ca/2012/07/websphere-application-server-v85-cross.html
Use Cross Component Trace (XCT) request Ids to see which log entries were generated by each request
http://www.youtube.com/watch?v=oXsfDoiNb5c
Use the IBM WebSphere Cross Component Trace Logviewer to view WebSphere Application Server log files
http://www.youtube.com/watch?v=qorwRZh4DgQ
High Performance Extensible Logging and Cross Component Trace Tech Video
https://www.youtube.com/watch?v=FKKfUdNux70
Configuring the memory leak policy
http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/topic/com.ibm.websphere.nd.doc/ae/ttrb_configmemleak.html
Memory leaks in Java Platform, Enterprise Edition applications
http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp?topic=%2Fcom.ibm.websphere.nd.doc%2Fae%2Fctrb_memleakdetection.html
Configuring the hang detection policy (full profile)
http://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.iseries.doc/ae/ttrb_confighangdet.html
Using sensitive log and trace guard
http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/topic/com.ibm.websphere.express.doc/ae/ctrb_sensitivelogtrace.html

Useful Links – Liberty Profile
37
Liberty Profile: Binary Logging
http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/topic/com.ibm.websphere.wlp.express.doc/ae/cwlp_HPELOverview.html
Binary logging for Liberty profile
http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.5.5.0/content/WASV855_Binary_Logging/player.html?dmuid=2013
0807101913462846
Timed operations for Liberty
http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.5.5.0/content/WASV855_Serviceability_TimedOperations/player.h
tml?dmuid=20130815082326115285
Lab: Liberty profile timed operations for JDBC
http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.5.5.0/content/WASv855_Liberty_Timed_Operations_Lab.pdf?dmui
d=20130807102148029081
http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.5.5.0/content/WASv855Labs_libtops.zip?dmuid=20130807102201
566285
Liberty Profile: Slow and hung request detection
http://www-01.ibm.com/support/knowledgecenter/was_beta_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/rwlp_requesttiming.html
Liberty profile: Event Logging
http://www-01.ibm.com/support/knowledgecenter/was_beta_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/rwlp_eventlogging.html
Liberty profile: Analytics
http://www-01.ibm.com/support/knowledgecenter/was_beta_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/rwlp_analytics.html
Liberty monitoring feature
http://www-01.ibm.com/support/knowledgecenter/SSD28V_8.5.5/com.ibm.websphere.wlp.nd.doc/ae/twlp_mon.html
Creating remote JMX connections in Liberty
https://developer.ibm.com/wasdev/docs/article_howto_remotejmx/
Configuring secure JMX connection to the Liberty profile
http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp?topic=%2Fcom.ibm.websphere.wlp.nd.doc%2Fae%2Ftwlp_admin_restconnector.html

Useful Links – Other
38
JVM
Writing Verbose GC to a Specified Log -Xverbosegclog in AIX, Linux and Windows
http://www-01.ibm.com/support/docview.wss?uid=swg21384096
PROBLEM DETERMINATION TOOLS
IBM Support Assistant
http://www.ibm.com/software/support/isa/
IBM Whole-system Analysis of Idle Time (WAIT)
http://wait.ibm.com
Memory Analyzer (MAT)
https://eclipse.org/mat/
EDUCATION AND TRAINING
IBM WebSphere Application Server V8.5.5 Problem Determination
http://www-304.ibm.com/jct03001c/services/learning/ites.wss/ca/en?pageType=course_description&courseCode=WA591CE
Support Technical Exchanges
http://www.ibm.com/support/entry/portal/Scheduled_tech_exchanges/Software/WebSphere/WebSphere_brand_support_(general)

Notices and Disclaimers
Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or
transmitted in any form without written permission from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with
IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been
reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM
shall have no responsibility to update this information. THIS document is distributed "AS IS" without any warranty, either express
or implied. In no event shall IBM be liable for any damage arising from the use of this information, including but not limited to, loss
of data, business interruption, loss of profit or loss of opportunity. IBM products and services are warranted according to the terms
and conditions of the agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without
notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are
presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual
performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products,
programs or services available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not
necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither
intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal
counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s
business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or
represent or warrant that its services or products will ensure that the customer is in compliance with any law.

Notices and Disclaimers (con’t)
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products in connection with this
publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM
products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those
products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products
to interoperate with IBM’s products. IBM expressly disclaims all warranties, expressed or implied, including but not
limited to, the implied warranties of merchantability and fitness for a particular purpose.
The provision of the information contained herein is not intended to, and does not, grant any right or license under any
IBM patents, copyrights, trademarks or other intellectual property right.
• IBM, the IBM logo, ibm.com, Bluemix, Blueworks Live, CICS, Clearcase, DOORS®, Enterprise Document
Management System™, Global Business Services ®, Global Technology Services ®, Information on Demand,
ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™,
PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®,
pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, SoDA, SPSS, StoredIQ, Tivoli®, Trusteer®,
urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International
Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names
might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at
"Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

Thank You
Your Feedback is
Important!
Access the InterConnect 2015
Conference CONNECT Attendee
Portal to complete your session
surveys from your smartphone,
laptop or conference kiosk.

Monitoring and analytics with was liberty

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (14)

Destaque

Destaque (20)

Semelhante a Monitoring and analytics with was liberty

Semelhante a Monitoring and analytics with was liberty (20)

Mais de sflynn073

Mais de sflynn073 (20)

Monitoring and analytics with was liberty