2. Agenda:
➢ Objective of performance monitoring
➢ How is this typically done ?
➢ Introduction to Performance Co-Pilot
➢ Component and Architecture
➢ Useful performance monitoring tools
➢ Demo
➢ Knowledge sources
3. Objective of performance monitoring
➢ Early detection of a ( potential ) problem.
➢ Rapidly drill down & pinpoint issue in specific program.
4. How is this typically done ?
➢ System log files ( rsyslog/syslog-ng/journald )
➢ Native performance monitoring tools ( top/iostat/vmstat/ps )
➢ Mixture of scripting languages ( bash/perl/python )
➢ Specific tools vary per platform
5. Why we need another Performance Monitoring
Tool ?
➢ Ability to monitor performance statistics of a local or remote
machines in a complex network.
➢ Ability to record and replay performance statistics.
➢ Ability to monitor multiple hosts performance statistics
concurrently.
➢ Customization of clients application to collect specific metrics.
➢ Performance regression detection.
➢ Evaluation of effects of an operating system upgrade etc.
6. Introduction to Performance Co-Pilot
➢ PCP is a system level performance monitoring and
performance management toolkit.
➢ PCP provides a range of services that may be used for
collection, monitoring and analysis of system metrics.
➢ The Performance Co-Pilot includes many facilities for creating
and replaying archive logs that capture performance
information.
➢ Cross platform support : Linux, Mac OS and Windows.
➢ End-to-end : Hardware, Core OS, services and
applications.
7. Introduction: Performance Co-Pilot
➢ Distributed architecture
Monitoring of local and remote nodes.
➢ Real-time or retrospective
Live system or archive.
➢ Pluggable
New agents system metrics within PCP.
8. Performance Co-Pilot - Components
Collectors
● Collect and export performance metrics
● Performance Metric Domain Agents (PMDA)
● Performance Metric Collection Daemon (PMCD)
Consumers
● Record, visualise, monitor and analyse performance data
● Consume data either in realtime or replay historical data from
archive logs
Note: Hosts may operate as collectors, consumers or both
● Multiple consumers may connect with one or more collectors
9. Performance Co-Pilot - Collectors
Performance Metric Domain Agents (PMDA)
● Extracts & exports metric data from a system component
● Communicates with pmcd on local system
Performance Metric Collection Daemon (PMCD)
● One pmcd process per host
● Coordinates handling of fetch requests between consumer
applications and agents
● Listens for connections from localhost & remote clients
● Authenticated & encrypted connection options
10. Performance Co-Pilot : Consumers
● pmlogger
● Utility to capture and store metrics exported by PMCD
● Concurrent logging of data from local and remote hosts
● Archive playback by other consumer tools (eg pmchart, pmval)
● Tools for log archival, log rotation etc
● Operates on data in realtime only
● pminfo - display PCP metrics available on a host or in an archive
● pmchart - GUI utility providing graphical display of PCP data
● pmstat - vmstat-like utility
● pmatop - top-like utility
● pmie - inference engine and alerting utility
● pmval - display a metric
13. Installation:
➢ To install pcp and pcp-gui package, type the following
command:
# yum install pcp pcpgui
“pcp” package provides a framework and services to support
system-level performance monitoring and performance
management.
“pcp-gui” package provides a visualization tools for the
Performance Co-Pilot toolkit.
14. Services:
To start pmlogger and pcp services, type the following command:
# chkconfig pmlogger on
# service pmlogger start
Starting pmlogger ...
# service pcp start
Starting pmcd ...
Starting pmlogger ...
To check the status of pcp service, type the following command:
# service pcp status
Checking for pmcd: running
Checking for pmlogger: running
Note: Bug 1044682 - pcp should use systemd
16. Configuration files:
Main configuration file for pcp:
/etc/pcp.conf
Main configuration file for pmcd:
/etc/pcp/pmcd/pmcd.conf
Default pmlogger config file:
/etc/pcp/pmlogger/config.default
PCP archive logging configuration/control:
/etc/pcp/pmlogger/control
Note: pmlogger is utility to capture and store metrics exported by PMCD
19. Performance monitoring tools:
➢ For all PCP monitoring tools, metrics values may come from a real-
time feed (i.e. from pmcd on some host), or from an archive log.
➢ Performance monitoring tools available in Performance Co-Pilot
(PCP).
pmstat
pmatop
pmcollectl
pmval
pmchart
pminfo
20. Performance monitoring tools:
pmstat - vmstat-like utility, intended to monitor system
performance at the highest level.
# pmstat
# pmstat t 1 T 3
# pmstat a
/var/log/pcp/pmlogger/localhost.localdomain/20140607.20.20.0
22. Performance monitoring tools:
pmcollectl - System-level performance monitoring utility that
records or displays specific operating system data for one or more
sets of subsystems.
Basic Systemlevel performance monitoring:
# pmcollectl
Subsystem Specific performance monitoring:
# pmcollectl sm
# pmcollectl sm –verbose
# pmcollectl smcdn –verbose
Record Mode:
# pmcollectl f perfdata c 3
Playback Mode:
# pmcollectl p perfdata
23. Performance monitoring tools:
pminfo - Display PCP metrics available on a host or in an archive.
Full list of all available metrics:
# pminfo F
Fetch and print values for all or specific metrics:
# pminfo f
# pminfo f <metric>
# pminfo F a
/var/log/pcp/pmlogger/localhost.localdomain/20140607.20.56.0
Eg:
# pminfo f proc.nprocs disk.dev.read filesys.free
Descriptions of all or specific metrics:
# pminfo T d
# pminfo T d <metric>
Eg:
# pminfo T d mem.util.cached
# pminfo T d proc.nprocs disk.dev.read filesys.free
24. Performance monitoring tools:
pmval - Dumps the current values for the named performance
metrics.
# pmval <metric>
Eg:
# pmval proc.nprocs
# pmval kernel.all.load
# pmval t 2sec s 4 kernel.percpu.cpu.idle
