IBM i Job wait accounting job watcher june 2012

IBM Power Systems

IBM i Job Watcher and Wait Accounting
Improve your Job Performance with Run-Wait Analysis

Dawn May - dmmay@us.ibm.com

© 2012 IBM Corporation

IBM Power Systems

Agenda

 What is Wait Accounting?
 Why is Wait Accounting important?
 How to Use Wait Accounting
– General information
– Collection Services
– Job Watcher

2 © 2012 IBM Corporation

IBM Power Systems

What is Wait Accounting?

Wait Accounting is the ability to determine what a job is
doing when it is not running
– When a job is not running, it is waiting
• But what is it waiting for?
– Waits may be normal, some waits are not normal
• Job Wait Accounting helps to determine if a wait condition is a problem


IBM i has instrumented most
of the wait conditions
– Wait information is collected by
Collection Services and Job Watcher


i Exclusive!


IBM Power Systems

What is Wait Accounting?
 Wait Accounting is used to understand what is happening
when a job is waiting
– Wait information is tracked for each job, thread and task on system

 A job spends its time in one of three states
– CPU
• Time spent dispatched to the processor, active/running
– CPU queuing
• Ready to be processed, waiting for a processor to become available
– Wait
• Waiting for something or someone, blocked or idle

http://ibmsystemsmag.blogs.com/i_can/2009/11/i-can-tell-you-why-youre-waiting.html


IBM Power Systems

Basics of Waiting
 Two basic types of waits
– Idle: waiting for a work request
• Typically not indicative of a problem
 E.g., waiting for the “Enter” key to be pressed on a 5250 display session
• If a problem, usually external to the machine
 E.g., slow arrival of work requests due to communications problem
• Possible, but not typical in batch jobs
 E.g., waiting for an entry to be placed on a data queue
– Blocked: waits that occur while performing a work request

 Blocked waits are the ones we want to take a closer look at
● “Outside of CPU usage and CPU queue time, blocked waits are the
reason jobs/threads take as long as they do to complete their work”


IBM Power Systems

Run/Wait Signature

Typical batch job run/wait signature
CPU CPU queue Wait
Elapsed time

Interactive job run/wait signature
Idle CPU CPU queue Wait
Elapsed time


IBM Power Systems

Why is Wait Accounting important?
 Wait time is easy to compute, but rarely scrutinized

For batch-type work:
Waits = Elapsed time – CPU time

For interactive-type work:
Waits = Elapsed time – CPU time – Idle time

▪ Historically, little instrumentation has existed to measure and
provide detail on waits
● i Wait Accounting provides the instrumentation to understand and
analyze waits


IBM Power Systems

Basic example
 Batch job with total run time of 6 hours
Run/wait signature
CPU CPU queue Wait
120 min 70 min 170 min
Elapsed time 6 hours (360 mins)

● Potential to run in 3 hrs 10 min if wait time could be eliminated

▪ Wait analysis and reduction can be a powerful and cost-
effective way to improve response time and throughput


IBM Power Systems

Detailing wait time
 Determine the components of time spent waiting

Elapsed time
CPU CPU queue Wait

Record
Disk reads Disk writes Journal
locks


IBM Power Systems

Detailing wait time - metrics
 Then get metrics related to these components of wait
time
Total Disk reads Disk writes Record Journal
count Locks
3,523 17,772 355 5,741
Total
time 42 sec 73 sec 45 sec 44 sec
0.012 sec 0.004 sec 0.126 sec 0.007 sec
Avg time
per wait

▪ Even at this level, we can already see possible questions to ask:
● How many of the reads are page faults? Could memory/pool changes help?
● What programs are causing reads? Could they be reduced or made async?
● What programs are causing writes? Could they be reduced or made async?
● What DB2 files are involved with the record locks?
● What files are being journaled? Are journals needed and optimally configured?


IBM Power Systems

Expected vs. Unexpected Waits

 Some waits are “expected” and others “unexpected”
– In most cases, “normal” and “abnormal” are more accurate descriptors
• Example: a record lock may be normal, but one that lasts for a very long
duration is abnormal
– Regardless of the type of wait, it is always better if wait time can be
minimized or eliminated
– There are a few block points on the system that are almost always
considered abnormal


IBM Power Systems

Holders versus Waiters

 IBM i keeps track of who is holding a resource, and if applicable, who is waiting
to access that resource

– A Holder is the job/thread/task that is holding the serialized resource

– A Waiter is the job/thread/task that wants to access the serialized resource

 IBM i also maintains call stacks for every job/thread/task

 The combination of wait buckets, information regarding holders/waiters, and call
stacks can provide a very powerful solution for analyzing wait conditions

 Job Watcher is the tool to accomplish this
– Collection services can tell you about waits, but not about holders/waiter nor call stacks


IBM Power Systems

Wait Accounting - “Wait Buckets”

 Licensed Internal Code (LIC) has identified points where waits
occur and assigned a numeric identifier to each point, also known
as a “block point”

 This level of detail poses challenges to efficiently evaluate the wait
time in a job
– Difficult to fully understand hundreds of block points
– Huge amounts of storage would be needed to track counts and times for every
block point in every job

 For this reason, block points are assigned to categories commonly
referred to as “wait buckets”

 “Mapping” refers to how block points are assigned to buckets


IBM Power Systems

32 Wait Buckets as Defined at 6.1 and 7.1
– Time dispatched on a CPU – Object lock contention
– CPU queuing – Ineligible waits
– Reserved – Main storage pool overcommitment
– Other waits – Classic Java user including locks
– Disk page faults – Classic Java JVM
– Disk non fault reads – Classic Java other
– Disk space usage contention – Socket accepts (idle)
– Disk op-start contention – Socket transmits
– Disk writes – Socket receives
– Disk other – Socket other
– Journaling – IFS
– Semaphore contention – PASE
– Mutex contention – Data queue receives
– Machine level gate serialization – Idle / waiting for work
– Seize contention – Synchronization Token contention
– Database record lock contention – Abnormal contention


IBM Power Systems

Wait Bucket Example

 Database Record Lock Contention – Bucket 16

– Several different causes for waits in this bucket
• Read
• Update
• Weak
• Transfer
• Check
• Conflict Exit


IBM Power Systems

Collection Services


As of 6.1 Collection Services collects all 32 wait buckets

When Create Performance Data (CRTPFRDTA) runs, data is
written to files
– QAPMJOBWT - wait bucket data file
• Information that accounts for the time a job, task, or thread spends waiting
●
Each job, task, or thread that consumed CPU during the interval
• Counts and clock times for each bucket
• Current wait: time waiting and bucket to be used when wait is satisfied
– QAPMJOBWTD - wait bucket description file
• Descriptions of the counter sets found in the above file – in unicode
– QAPMJOBWTG – wait conditions not available in the QAPMJOBWT file
• One record for each job, task, or thread that did not consume CPU during the interval

Note: The Information Center also describes ‘buckets’ as ‘groups’ or ‘sets’


IBM Power Systems

Understanding “Time Dispatched on a CPU”
Time dispatched on a CPU (Bucket 1)
– Thread or task has been assigned to a processor and is NOT waiting
– Complicated by certain features
• Hardware Multi Threading (HMT)
●
Allows multiple threads/tasks to be assigned to a single physical processor
●
Causes bucket 1 time to be greater than actual CPU time
• Background assisting tasks
●
Promote their CPU usage back into the client job/thread
●
Causes client thread’s bucket 1 time to be smaller than measured CPU time
• LPAR shared/partial processors
●
Bucket 1 records time dispatched to the virtual processor
●
Bucket 1 time may be greater than CPU time because it may include time the
thread/task is waiting for the physical processor behind the virtual processor

Bucket 1 does NOT equal CPU Time

IBM Power Systems

Understanding “CPU Queuing”
CPU Queuing (Wait Bucket 2)
– Thread or task has been assigned to a processor and is waiting for the
CPU to become available

• Too much work on the partition causing threads to need to wait for the
processors

• Spiky workloads

• Shared processors

Latency due to hypervisor sharing the physical processors among
multiple partitions


IBM Power Systems

Waits that Applications may use


Disk waits

Semaphores, Mutexes, Synchronization Tokens

Journaling

Database record locks

Object locks

Sockets


IBM Power Systems

Collection Services vs Job Watcher

 Collection Services and Job Watcher both collect wait
information
– Both provide graphs that show waits
– Collection Services runs by default in 6.1 and later
– Job Watcher data is generally collected when additional information is
necessary to analyze a problem

 Job Watcher graphs have better drill-down capabilities
– Display graphs that have the data sorted by wait types
– Collection Services can accomplish this via table views

 Job Watcher can also collect call stacks and SQL
statements
– Provides additional information for detailed analysis

IBM Power Systems

IBM Systems Director Navigator for i
Job Watcher

 Job Watcher GUI is included with IBM i
Performance Tools, Job Watcher Feature

 From GO LICPGM:

– 5761PT1 *COMPATIBLE IBM Performance Tools for i5/OS
– 5761PT1 *COMPATIBLE Performance Tools - Manager Feature
– 5761PT1 *COMPATIBLE Performance Tools - Agent Feature
 5761PT1 *COMPATIBLE Performance Tools - Job Watcher


IBM Power Systems

IBM Systems Director Navigator for i

 IBM Systems Director Navigator for i is the Web console for managing
IBM i

– Much of the function that exists in System i Navigator, but with a browser user
interface


IBM Power Systems

Prerequisites
 Users need to be authorized to use the Investigate Data and
Collections performance tasks

 Include users on the QPMCCDATA authorization list

Edit Authorization List

Object . . . . . . . : QPMCCDATA Owner . . . . . . . : QSYS
Library . . . . . : QSYS Primary group . . . : *NONE

Type changes to current authorities, press Enter.

Object List
User Authority Mgt
*PUBLIC *EXCLUDE
QSYS *ALL X
PDI01 *USE
PDI02 *USE
PDI03 *USE
PDI04 *USE
PDI05 *USE
PDI06 *USE
PDI07 *USE
PDI08 *USE
PDI09 *USE
More...


IBM Power Systems

Prerequisites, continued
 To see the Job Watcher definitions through the GUI, users need
authority to the QAPYJWDFN file in QUSRSYS
– File is shipped with *PUBLIC *EXCLUDE


IBM Power Systems

How do I run Job Watcher?


IBM Power Systems

IBM-Supplied Job Watcher Definitions
 Several pre-defined Job Watcher definitions are available
– The main difference is the sample intervals
• Q1SEC* – 1 second intervals with call stacks, another to include SQL
• Q10SEC* –10 second intervals with call stacks, another to include SQL
• Q5SEC* – 5 second intervals with call stacks, another to include SQL


IBM Power Systems

Adding a Job Watcher Definition


IBM Power Systems

Starting Job Watcher


IBM Power Systems

How Do I Run Job Watcher with the Commands?

 CL Commands also available starting with 6.1
– Add Job Watcher Definition (ADDJWDFN) to define the collection
• Identifies the performance data that is to be collected
– Remove Job Watcher Definition (RMVJWDFN) to remove a definition
(Note: Job Watcher Definitions can only be displayed through the GUI)

– Start Job Watcher (STRJW) to start the collection
– End Job Watcher (ENDJW) to end the collection (optionally)

 Prior to 6.1, the iDoctor product is required to collect Job Watcher
data


IBM Power Systems

Job Watcher Definition Parameters
 Collection interval
– Interval between retrieval of job/task data
– Considerations for frequency of sampling and amount of data collected
 Additional data categories
– Additional types of data to include in the collection
• For example, call stacks, Java, sockets and TCP data
– Allows granularity on the data collected and frequency
 Wait-based call stack data
– A special case where call stacks can be collected for certain jobs and/or tasks
which have been in an abnormal or conflict wait
 Job name
– The jobs that will be included in the collection
 Task name
– The tasks that will be included in the collection
 TDE number
– The Task Dispatching Elements (TDE) that will be included in the collection
 Current user profile
– The current user profile of the jobs which will be included in the collection


IBM Power Systems

Job Watcher Definition Parameters
 Subsystem
– The subsystem of the jobs which will be included in the collection
 Current storage pool
– The current system pool of the jobs and/or tasks which will be included in the
collection
 Force record write
– When data records are written to the database files
– At the end of the interval is necessary for real time analysis
 Include inactive jobs/tasks
– Whether detailed data for inactive jobs and tasks will be collected during the
first interval
 To file ASP threshold
– Refers to the ASP where the Job Watcher database files are located
– The maximum percent of this ASP that can be consumed
 System ASP threshold
 The maximum percent of the system ASP that can be consumed

IBM Power Systems

Job Watcher Definition - Conditional parameters

Provides a way to limit the size of the data collection
– Can specify parameters to conditionally collect data
• Average, Percentage, Rate, Direct comparisons
– Can specify the type of conditional collection
• Per interval checks in every interval
● Data will be collected for intervals in which the condition was satisfied

• Trigger checks until the condition is satisfied
● Once the condition is met, data will unconditionally be collected

• Until Met collects data unconditionally until the condition is met
● Once the condition is met, the collection ends

– Timeout option
• How long the collection should run without the condition being met
– Consecutive occurrence count
– History size
– User exit program
• Program that will be called at the time the condition is satisfied


IBM Power Systems

Start Job Watcher Parameters
 Definition
– Name of the definition that identifies the collection parameters
 Collection
– The name you want the collection to be known by
 Library
– Library where the database files will be created
 Text description
– Description associated with the database file members
 Replace data
– Whether to overwrite existing database file members with the specified name
 Collection Interval
– Can override what is specified on the definition
 End collection
– Determines when data collection will end
– Multiple end options may be specified
• Elapsed time, storage consumed, number of intervals
● Whichever occurs first will end the collection
 Job name, Task name, TDE number
– Can override what is specified on the definition

IBM Power Systems

Managing Job Watcher Collections


IBM Power Systems

Managing Job Watcher Collections …


IBM Power Systems

How do I analyze Job Watcher data?
 Look for trends in the data

 Look for presence of waits
– Drill down into wait details

 Display call stacks for running or waiting jobs


IBM Power Systems

Job Watcher Content Package


IBM Power Systems

CPU Utilization and Waits Overview
 Both Collection Services and Job Watcher have a “CPU
Utilization and Waits Overview” graph as a general
starting point for wait analysis

CPU Utilization and
Waits Overview

IBM Power Systems

IBM Systems Lab Services and Training
Mainframe Systems

Our Mission and Profile
 Support the IBM Systems Agenda and accelerate the adoption of new
Power Systems
products and solutions
 Maximize performance of our clients’ existing IBM systems
 Deliver technical training, conferences, and other services tailored to System x & Bladecenter
meet client needs
 Team with IBM Service Providers to optimize the deployment of IBM
solutions (GTS, GBS, SWG Lab Services and our IBM Business System Storage
Partners)

Our Competitive Advantage
IT Infrastructure Optimization
 Leverage relationships with the IBM development labs to build deep
technical skills and exploit the expertise of our developers
 Combined expertise of Lab Services and the Training for Systems team Data Center Services
 Skills can be deployed worldwide to assure all client needs can be met

Successful worldwide history:
Training Services
17 years in Americas, 9 years in Europe/Middle East/Africa,
5 years in Asia Pacific

www.ibm.com/systems/services/labservices stgls@us.ibm.com

IBM Power Systems

IBM Systems Lab Services and Training Power Services
Key Offerings Americas, WW Contacts
 High Availability Services on Power Systems (including
Advanced Copy Services for PowerHA™ on IBM i)
Mark Even
even@us.ibm.com, 507-253-1313
 Systems Director Services IBM i

 PowerCare Services Frank Kriss
 Performance and Scalability services (including system, kriss@us.ibm.com, 507-253-1354
application, and database tuning) IBM i, High Availability

 Virtualization Services for AIX® on Power Systems™ Karen Anderson
 Application and database modernization consulting (SOA kanders@us.ibm.com, 972-561-6337
implementation) IBM i Vouchers

 Linux® on Power consulting, custom application Stephen Brandenburg
development, implementation, and optimization services sbranden@us.ibm.com, 301-803-6199
PowerVouchers, Virtualization Program, AIX
 Security on Power consulting and implementation services SWOT/SWAT, AIX
 System consolidation and migration service

Allen Johnston
High Performance Computing consulting and allenrj@us.ibm.com, 704-340-9165
implementation services PowerCare
 SAP® on IBM i consulting
Cindy Mestad
 Power Blades on BladeCenter (including VIOS on i and cindysm@us.ibm.com, 507-253-6143
blades running IBM i implementation) Power Performance and Scalability Center
 Smart Analytics services (including DB2® Web Query
implementation and consulting)
 Public, private, customized and self-paced virtual training
 Power Systems Technical University
www.ibm.com/systems/services/labservices stgls@us.ibm.com © 2012 IBM Corporation

IBM Power Systems

Performance and Scalability Services
 The IBM i Performance and Scalability Services Center can provide facilities and
hardware IN ROCHESTER to assist you in testing hardware or software changes

 “Traditional” benchmarks
 Release-to-release upgrades
 Assess application performance when migrating to a new release of IBM I
 Stress test your system
 Determine impact of application changes
 Proofs of Concept (e.g. HA alternatives; SSD analysis, external storage, etc.)
 Evaluate application scalability
 Capacity planning

 … all with the availability of Lab Services IBM i experts and development personnel

 To request any of these services, submit at:

• http://www-03.ibm.com/systems/services/labservices/psscontact.html


IBM Power Systems

Back Up and Reference Information


IBM Power Systems

Reference Information for Job Watcher
 iDoctor versus IBM i 6.1 Performance Tools
– http://www.ibmsystemsmag.com/i5/october08/trends/21990p1.aspx

 “Web Power”
– http://www.ibmsystemsmag.com/i5/november08/administrator/22431p1.aspx

 Introduction to Job Watcher Green Screen Commands
– http://www.ibmsystemsmag.com/i5/november08/tipstechniques/22521p1.aspx

 End-to-End Performance Management on IBM i – a Redbooks
publication
– http://www.redbooks.ibm.com/redbooks/pdfs/sg247808.pdf

 i Can … Tell You Why You Are Waiting
– http://ibmsystemsmag.blogs.com/i_can/2009/11/i-can-tell-you-why-youre-waiting.html


IBM Power Systems

Special notices
This document was developed for IBM offerings in the United States as of the date of publication. IBM may not make these offerings available in
other countries, and the information is subject to change without notice. Consult your local IBM business contact for information on the IBM
offerings available in your area.
Information in this document concerning non-IBM products was obtained from the suppliers of these products or other public sources. Questions
on the capabilities of non-IBM products should be addressed to the suppliers of those products.
IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give
you any license to these patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY
10504-1785 USA.
All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives
only.
The information contained in this document has not been submitted to any formal IBM test and is provided "AS IS" with no warranties or
guarantees either expressed or implied.
All examples cited or described in this document are presented as illustrations of the manner in which some IBM products can be used and the
results that may be achieved. Actual environmental costs and performance characteristics will vary depending on individual client configurations
and conditions.
IBM Global Financing offerings are provided through IBM Credit Corporation in the United States and other IBM subsidiaries and divisions
worldwide to qualified commercial and government clients. Rates are based on a client's credit rating, financing terms, offering type, equipment
type and options, and may vary by country. Other restrictions may apply. Rates and offerings are subject to change, extension or withdrawal
without notice.
IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies.
All prices shown are IBM's United States suggested list prices and are subject to change without notice; reseller prices may vary.
IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.
Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are
dependent on many factors including system hardware configuration and software design and configuration. Some measurements quoted in this
document may have been made on development-level systems. There is no guarantee these measurements will be the same on generally-
available systems. Some measurements quoted in this document may have been estimated through extrapolation. Users of this document
should verify the applicable data for their specific environment.

Revised September 26, 2006


IBM Power Systems

Special notices (cont.)
IBM, the IBM logo, ibm.com AIX, AIX (logo), AIX 6 (logo), AS/400, Active Memory, BladeCenter, Blue Gene, CacheFlow, ClusterProven, DB2, ESCON, i5/OS, i5/OS
(logo), IBM Business Partner (logo), IntelliStation, LoadLeveler, Lotus, Lotus Notes, Notes, Operating System/400, OS/400, PartnerLink, PartnerWorld, PowerPC, pSeries,
Rational, RISC System/6000, RS/6000, THINK, Tivoli, Tivoli (logo), Tivoli Management Environment, WebSphere, xSeries, z/OS, zSeries, AIX 5L, Chiphopper, Chipkill,
Cloudscape, DB2 Universal Database, DS4000, DS6000, DS8000, EnergyScale, Enterprise Workload Manager, General Purpose File System, , GPFS, HACMP,
HACMP/6000, HASM, IBM Systems Director Active Energy Manager, iSeries, Micro-Partitioning, POWER, PowerExecutive, PowerVM, PowerVM (logo), PowerHA, Power
Architecture, Power Everywhere, Power Family, POWER Hypervisor, Power Systems, Power Systems (logo), Power Systems Software, Power Systems Software (logo),
POWER2, POWER3, POWER4, POWER4+, POWER5, POWER5+, POWER6, POWER7, pureScale, System i, System p, System p5, System Storage, System z, Tivoli
Enterprise, TME 10, TurboCore, Workload Partitions Manager and X-Architecture are trademarks or registered trademarks of International Business Machines
Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a
trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such
trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark
information" at www.ibm.com/legal/copytrade.shtml

The Power Architecture and Power.org wordmarks and the Power and Power.org logos and related marks are trademarks and service marks licensed by Power.org.
UNIX is a registered trademark of The Open Group in the United States, other countries or both.
Linux is a registered trademark of Linus Torvalds in the United States, other countries or both.
Microsoft, Windows and the Windows logo are registered trademarks of Microsoft Corporation in the United States, other countries or both.
Intel, Itanium, Pentium are registered trademarks and Xeon is a trademark of Intel Corporation or its subsidiaries in the United States, other countries or both.
AMD Opteron is a trademark of Advanced Micro Devices, Inc.
Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries or both.
TPC-C and TPC-H are trademarks of the Transaction Performance Processing Council (TPPC).
SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPEC OMP, SPECviewperf, SPECapc, SPEChpc, SPECjvm, SPECmail, SPECimap and SPECsfs are
trademarks of the Standard Performance Evaluation Corp (SPEC).
NetBench is a registered trademark of Ziff Davis Media in the United States, other countries or both.
AltiVec is a trademark of Freescale Semiconductor, Inc.
Cell Broadband Engine is a trademark of Sony Computer Entertainment Inc.
InfiniBand, InfiniBand Trade Association and the InfiniBand design marks are trademarks and/or service marks of the InfiniBand Trade Association.
Other company, product and service names may be trademarks or service marks of others.

Revised February 9, 2010


IBM i Job wait accounting job watcher june 2012

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (9)

Similar to IBM i Job wait accounting job watcher june 2012

Similar to IBM i Job wait accounting job watcher june 2012 (20)

More from COMMON Europe

More from COMMON Europe (20)

Recently uploaded

Recently uploaded (20)

IBM i Job wait accounting job watcher june 2012