1. Monitoring and Tuning the System
  2. Working With Performance Co-Pilot

3 Working With Performance Co-Pilot

Performance Co-Pilot (PCP) collects OS and network metrics that you can use to diagnose performance issues. PCP provides a monitor host that you can use to send requests for metrics and logs to a pair of collector host services that are installed on each Oracle Linux system that you monitor.

Note:

PCP also provides metrics and other performance-related information that were previously provided by the Oracle OSWatcher Black Box (OSWbb) feature, as described in Oracle Linux 8: Monitoring and Tuning the System.

Installing PCP

  1. Enable the ol9_appstream yum repository on the system.

    For more information, see Oracle Linux: Managing Software on Oracle Linux.

  2. Install the pcp, pcp-system-tools, and pcp-gui packages by using the dnf command: 

    sudo dnf install pcp pcp-system-tools pcp-gui

  3. Enable and start the Performance Metrics Collector Daemon (pmcd) and Performance Metrics Logger (pmlogger) collector host services: 

    sudo systemctl enable --now pmcd pmlogger

Stopping PCP

To temporarily halt data collection, stop the Performance Metrics Collector Daemon (pmcd) and Performance Metrics Logger (pmlogger) collector host services: 

sudo systemctl stop pmcd pmlogger

To halt data collection for an indefinite period and ensure that that they don't start again automatically when the system boots, fully disable them:

sudo systemctl disable --now pmcd pmlogger

For more information about masking and unmasking services to prevent scripts from restarting disabled system services, see Oracle Linux 9: Managing Core System Configuration.

Reviewing Information Gathered by PCP

If the pcp-oracle-conf package is installed then the only metrics collected by the pmlogger service are those listed in the /var/lib/pcp/config/pmlogger/config.ora configuration file.

If PCP has been installed without the pcp-oracle-conf package, review the /var/lib/pcp/config/pmlogger/config.default configuration file instead.

You can change the frequency with which those metrics are collected in the same configuration file. For example, to increase the frequency from each minute to every 5 seconds, revise the file as follows:

...
# It is safe to make additions from here on ...
#

log mandatory on every 5 seconds {
  filesys.free
  filesys.used
  ...
}

All the archives that the pmlogger service generates are stored in the /var/log/pcp/pmlogger/hostname directory. For more information, see the pmlogconf(1) manual page.

To verify the PCP configuration at the time that pmlogger collected specific performance metrics, use the pcp command: 

sudo pcp -a 20220321.0.xz

Using PCP Monitor Host to Analyze Performance Metrics

Different commands are available for viewing data about the system's performance. The commands you use depend on whether you want to view the data in real time or from the logs. The following sections provide sample commands to display performance information.

For more information about the commands and parameters to use to view performance metrics, see their respective manual pages. See also https://pcp.io/docs/guide.html for more examples of how to display the system's different performance metrics.

Review Live Performance Metrics in Real Time

To monitor all the outgoing metrics from the eth0 network interface in real time, use the pmrep command: 

sudo pmrep -i eth0 -v network.interface.out

To monitor live hard drive operations for each partition with a two second interval, use the pmval command: 

sudo pmval -t 2sec -f 3 disk.partitions.write

Review Recorded Performance Metrics

All the archives that the pmlogger service generates are stored in the /var/log/pcp/pmlogger/hostname directory. Navigate to this directory to review the archived performance metrics.

The pmstat command can provide system performance metrics in a format similar to that produced by the sar command. For example, to review performance metrics averaged over 10 minute interval between 09:00 and 10:00 on a specific date: 

sudo pmstat -t 10m -S @09:00 -T @10:00 -a 20220321.0.xz

To compare the metrics between two time periods, use the pmdiff command. For example, to compare the metrics between 02:00 and 03:00 on one day to the metrics between 09:00 and 10:00 on a different day: 

sudo pmdiff -S @02:00 -T @03:00 -B @09:00 -E @10:00 20220321.0.xz 20220320.0.xz

Review Details About Recorded Performance Metrics

To review detailed information about a specific metric, use the pminfo command. For example, to review details about free memory: 

sudo pminfo -df mem.freemem -a 20220321.0.xz

Validate System Status When Performance Metrics Were Captured

To verify the host, timezone, and time period that an archive containing performance metrics contains, use the pmdumplog command: 

sudo pmdumplog -L 20220321.0.xz

To review a list of every enabled performance metric, use the pminfo command: 

sudo pminfo -a 20220321.0.xz

Running dstat With Performance Co-Pilot

The dstat utility in previous Oracle Linux releases is no longer being actively developed. Instead, it's implemented with Performance Co-Pilot (PCP) to perform many of the same functionalities to diagnose system performance.

This tool is immediately available with the installation of the PCP packages as described in Installing PCP.

To obtain information on the usage of the new dstat utility, type: 

pcp dstat -h
Usage: pcp-dstat [-afv] [options...] [delay [count]]
Versatile tool for generating system resource statistics

Dstat options:
  -c, --cpu             enable cpu stats
     -C 0,3,total          include cpu0, cpu3 and total
  -d, --disk            enable disk stats
     -D total,sda          include sda and total
...

By default, running the command without any other options shows you statistics about CPU, disk, network, page, and system use. 

pcp dstat
You did not select any stats, using -cdngy by default.
----total-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw 
  0   0 100   0   0|   0     0 | 198B  719B|   0     0 | 156   254 
  0   0 100   0   0|   0    12k|  66B  302B|   0     0 | 160   264 
  0   0  99   0   0|   0     0 | 132B  384B|   0     0 | 136   219 
...

As with the previous iteration of the tool, pcp dstat generates a running list of metrics or statistics in real time. To stop the process, type Ctrl-c.

You can narrow the information output by using different options that are available for the command. For example, to display the statistics only of CPU 1, you would type:

pcp dstat -c -C 1,total
-----cpu1-usage---------total-usage----
usr sys idl wai stl:usr sys idl wai stl
  0   0 100   0   0:  0   0 100   0   0
  1   0 100   0   0:  0   0  99   0   0
  0   0 100   0   0:  0   0 100   0   0
...

Similarly, to display only network statistics of a specific interface, such as ens3, and including totals, you would type:

pcp dstat -n -N ens3,total
--net/ens3---net/total-
 recv  send: recv  send
  66B  350B:  66B  350B
  66B  190B:  66B  190B
  66B  198B:  66B  198B
  66B  198B:  66B  198B
...

To store any statistics that are being gathered into a file for later review, include the -o outputfile option in the command.

For example, to collect network statistics and save the information in a log, type:

pcp dstat -n -f -o /tmp/netstat-log

Use the -f option to review full information. 

--net/ens3-----net/lo--
 recv  send: recv  send
  66B  358B:   0     0 
  66B  174B:   0     0 
  66B  190B:   0     0 
 341B  419B:   0     0 
  66B  190B:   0     0 
  66B  190B:   0     0 
  66B  190B:   0     0 ^C

Thereafter, to view the log, type:

cat /tmp/netstat-log
...
"Host:","hostname",,,,"User:","user"
"Cmdline:","pcp-dstat -n -f -o /tmp/netstat-log",,,,"Date:","date"
"net/ens3",,"net/lo",
"net/ens3:recv","net/ens3:send","net/lo:recv","net/lo:send"
65.934,357.641,0,0
66.000,173.999,0,0
66.000,190.001,0,0
340.992,418.991,0,0
66.001,190.004,0,0
66,190,0,0
66.000,189.999,0,0

For more information about pcp dstat, see the pdp-dstat(1) manual page.