Note:
- This tutorial is available in an Oracle-provided free lab environment.
- It uses example values for Oracle Cloud Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your cloud environment.
Work With Performance Co-Pilot on Oracle Linux
Introduction
Performance Co-Pilot (PCP) collects OS and network metrics that you can use to diagnose performance issues. PCP is a distributed architecture with collector hosts and monitor hosts.
PCP collector hosts are hosts being monitored. These hosts require a Performance Metrics Domain Agent (PMDA) which is responsible for collecting performance measurements. All PMDAs are controlled by the Performance Metrics Collector Daemon (pmcd
) on the same hosts.
PCP monitor hosts display data collected from hosts, or archives, that have the PCP collector installed. Many monitor tools are included with the core PCP package.
Objectives
In this lab, you will learn:
- Configure one Oracle Linux host as a PCP collector and monitor
- Use PCP monitoring tools on a single host
- Configure remote monitoring of a 2nd Oracle Linux host
- Monitor performance with pmchart GUI
Prerequisites
- Two (2) Oracle Linux 9 systems.
Setup Lab Environment
Note: When using the free lab environment, see Oracle Linux Lab Basics for connection and other usage instructions.
-
Open a terminal and connect via SSH to the ol-node-01 instance.
ssh oracle@<ip_address_of_ol-node-01>
-
Install the
pcp
,pcp-system-tools
, andpcp-gui
packages on the ol-node-01 instance.sudo dnf install pcp pcp-system-tools pcp-gui
The
pcp
package provides the basic PCP tools and services for collecting performance data. Thepcp-system-tools
andpcp-gui
packages are optionally installed on the monitoring host to extend the set of monitoring tools from the basepcp
package. -
Enable and start the Performance Metrics Collector Daemon (
pmcd
) and the Performance Metrics Logger (pmlogger
) collector host services.sudo systemctl enable --now pmcd pmlogger
Use PCP monitoring tools on a single host
-
Run the
pcp
utility on the ol-node-01 instance.sudo pcp
The default output summarizes the PCP installation on the local hosts.
Example Output:
[oracle@ol-node-01 ~]$ sudo pcp Performance Co-Pilot configuration on ol-node-01: platform: Linux ol-node-01 5.15.0-204.147.6.2.el9uek.x86_64 #2 SMP Thu Mar 7 22:09:50 PST 2024 x86_64 hardware: 4 cpus, 1 disk, 1 node, 31828MB RAM timezone: GMT services: pmcd pmcd: Version 6.2.0-1, 12 agents, 6 clients pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm jbd2 dm openmetrics pmlogger: primary logger: /var/log/pcp/pmlogger/ol-node-01/20240508.12.41 pmie: primary engine: /var/log/pcp/pmie/ol-node-01/pmie.log
-
Alternatively, you can pass a command to
pcp
to run, for example,pcp free
.sudo pcp free
Example Output:
[oracle@ol-node-01 ~]$ sudo pcp free total used free shared buff/cache available Mem: 32592652 1357328 23443732 27712 7791592 30732656 Swap 6291452 0 6291452
When
pcp
is invoked with a command to run, for examplefree
, it runspcp-free
which is located in the/usr/libexec/pcp/bin/
directory. -
List commands in the
/usr/libexec/pcp/bin/
directory that begin withpcp-
.sudo ls /usr/libexec/pcp/bin/pcp-*
Example Output:
[oracle@ol-node-01 ~]$ sudo ls /usr/libexec/pcp/bin/pcp-* /usr/libexec/pcp/bin/pcp-atop /usr/libexec/pcp/bin/pcp-pidstat /usr/libexec/pcp/bin/pcp-atopsar /usr/libexec/pcp/bin/pcp-ps /usr/libexec/pcp/bin/pcp-buddyinfo /usr/libexec/pcp/bin/pcp-python /usr/libexec/pcp/bin/pcp-dmcache /usr/libexec/pcp/bin/pcp-reboot-init /usr/libexec/pcp/bin/pcp-dstat /usr/libexec/pcp/bin/pcp-shping /usr/libexec/pcp/bin/pcp-free /usr/libexec/pcp/bin/pcp-slabinfo /usr/libexec/pcp/bin/pcp-htop /usr/libexec/pcp/bin/pcp-ss /usr/libexec/pcp/bin/pcp-iostat /usr/libexec/pcp/bin/pcp-summary /usr/libexec/pcp/bin/pcp-ipcs /usr/libexec/pcp/bin/pcp-tapestat /usr/libexec/pcp/bin/pcp-lvmcache /usr/libexec/pcp/bin/pcp-uptime /usr/libexec/pcp/bin/pcp-meminfo /usr/libexec/pcp/bin/pcp-verify /usr/libexec/pcp/bin/pcp-mpstat /usr/libexec/pcp/bin/pcp-vmstat /usr/libexec/pcp/bin/pcp-netstat /usr/libexec/pcp/bin/pcp-zoneinfo /usr/libexec/pcp/bin/pcp-numastat
-
Explore by running
pcp
and pass different commands, for example, runpcp atop
.Note: Some of these next command examples require you to press
CTRL-C
to exit.sudo pcp atop
-
Run
pcp atopsar
.sudo pcp atopsar
-
Run
pcp buddyinfo
.sudo pcp buddyinfo
-
Run
pcp dstat
.sudo pcp dstat
Note: Each of these
pcp-
commands has aman
page which describes the command and provides usage. -
View the
man
page for differentpcp-
commands, for example, runman pcp-atop
.man pcp-atop
-
Run
man pcp-atopsar
.man pcp-atopsar
-
Run
man pcp-buddyinfo
.man pcp-buddyinfo
-
Run
man pcp-dstat
.man pcp-dstat
-
In addition to the
pcp-
commands in the/usr/libexec/pcp/bin/
directory, this directory contains many other files provided by thepcp
package. Runls /usr/libexec/pcp/bin/
to view all of the files in this directory.ls /usr/libexec/pcp/bin
Example Output:
[oracle@ol-node-01 ~]$ ls /usr/libexec/pcp/bin chkhelp pcp-numastat pmgetopt pmlogger_rewrite discover pcp-pidstat pmhostname pmlogredact find-filter pcp-ps pmie_check pmlogreduce install-sh pcp-python pmie_daily pmlogrewrite mkaf pcp-reboot-init pmie_dump_stats pmnewlog newhelp pcp-shping pmie_email pmnsadd pcp-atop pcp-slabinfo pmie_farm pmnsdel pcp-atopsar pcp-ss pmiestatus pmnsmerge pcp-buddyinfo pcp-summary pmie_webhook pmpause pcp-dmcache pcp-tapestat pmlock pmpost pcp-dstat pcp-uptime pmlogconf pmproxy pcp-free pcp-verify pmlogextract pmsignal pcp-htop pcp-vmstat pmlogger pmsleep pcp-iostat pcp-zoneinfo pmlogger_check pmsnap pcp-ipcs pmcd pmlogger_daily pmwtf pcp-lvmcache pmcd_wait pmlogger_daily_report runaspcp pcp-meminfo pmconfig pmlogger_farm telnet-probe pcp-mpstat pmcpp pmlogger_janitor pcp-netstat pmfind_check pmlogger_merge
Note: A
man
page also exists for many of these commands.
Use PCP monitoring tools from /usr/bin
on a single host
The pcp
package installs many performance monitoring utilities in the /usr/bin
directory.
-
Use the
rpm
command to list files included in the PCP package that are installed in the/usr/bin
directory.rpm -ql pcp | grep /usr/bin
Example Output:
[oracle@ol-node-01 ~]$ rpm -ql pcp | grep /usr/bin /usr/bin/dbpmda /usr/bin/pcp /usr/bin/pmafm /usr/bin/pmdate /usr/bin/pmdiff /usr/bin/pmdumplog /usr/bin/pmevent /usr/bin/pmfind /usr/bin/pmgenmap /usr/bin/pmie /usr/bin/pmie2col /usr/bin/pmieconf /usr/bin/pmiectl /usr/bin/pminfo /usr/bin/pmjson /usr/bin/pmlc /usr/bin/pmlogcheck /usr/bin/pmlogconf /usr/bin/pmlogctl /usr/bin/pmlogdump /usr/bin/pmlogextract /usr/bin/pmlogger /usr/bin/pmloglabel /usr/bin/pmlogmv /usr/bin/pmlogpaste /usr/bin/pmlogredact /usr/bin/pmlogreduce /usr/bin/pmlogrewrite /usr/bin/pmlogsize /usr/bin/pmlogsummary /usr/bin/pmprobe /usr/bin/pmpython /usr/bin/pmrepconf /usr/bin/pmsearch /usr/bin/pmseries /usr/bin/pmsocks /usr/bin/pmstat /usr/bin/pmstore /usr/bin/pmtrace /usr/bin/pmval
-
Explore by running of few of these commands, for example, run
pminfo -t
to list all of the enabled performance metrics and include a short description of each.pminfo -t
Example Output:
[oracle@ol-node-01 ~]$ pminfo -t jbd2.njournals [Count of active JBD2 (Journal Block Device v2) devices] jbd2.transaction.count [Total transactions committed per journal] jbd2.transaction.requested [Total journal transactions requested per journal] jbd2.transaction.max_blocks [Maximum transaction blocks (buffers) per journal] jbd2.transaction.total.blocks [Total transaction blocks per journal] jbd2.transaction.total.blocks_logged [Total logged blocks per journal] jbd2.transaction.total.handles [Total handle count per journal] jbd2.transaction.total.time.waiting [Total time waiting per journal] jbd2.transaction.total.time.request_delay [Total request delay per journal] jbd2.transaction.total.time.running [Total running time per journal] ... disk.wwid.avg_rqsz [average I/O request size for both reads and writes] disk.wwid.r_avg_rqsz [average I/O request size for reads to the device] disk.wwid.w_avg_rqsz [average I/O request size for writes to the device] disk.wwid.d_avg_rqsz [average I/O request size for discards to the device] disk.wwid.util [percentage of time device was busy processing requests] proc.psinfo.age [time in seconds since process was started] proc.io.total_bytes [total bytes read and written by process] proc.hog.cpu [average percentage CPU utilization of each process] proc.hog.mem [sum of resident and swapped memory used by the process] proc.hog.disk [average I/O rate of each process since it was started]
Note: Some of these next command examples require you to press
CTRL-C
to exit. -
Run the
pmrep
command to view network interface information.pmrep network.interface.hw_addr
Example Output:
[oracle@ol-node-01 ~]$ pmrep network.interface.hw_addr n.i.hw_addr n.i.hw_addr lo ens3 00:00:00:00 02:00:17:0b 00:00:00:00 02:00:17:0b 00:00:00:00 02:00:17:0b ^C
-
Run the
pmrep
command to view outgoing metrics from the ens3 network interface in real time.pmrep -i ens3 -v network.interface.out
Example Output:
[oracle@ol-node-01 ~]$ pmrep -i ens3 -v network.interface.out n.i.o.bytes n.i.o.packets n.i.o.errors n.i.o.drops n.i.o.fifo n.i.o.carrier n.i.o.compressed ens3 ens3 ens3 ens3 ens3 ens3 ens3 byte/s count/s count/s count/s count/s count/s count/s N/A N/A N/A N/A N/A N/A N/A 279.594 1.997 0.000 0.000 0.000 0.000 0.000 ^C
-
Use the
pmval
command to view live disk write operations per partition.pmval disk.partitions.write
Example Output:
[oracle@ol-node-01 ~]$ pmval disk.partitions.write metric: disk.partitions.write host: ol-node-01 semantics: cumulative counter (converting to rate) units: count (converting to count / sec) samples: all sda1 sda2 sda3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.9993 ^C
Run PCP tools against PCP archives
The pmlogger
service generates archives of performance metrics. PCP archives are located in /var/log/pcp/pmlogger/<hostname>
.
-
View the archives in the
/var/log/pcp/pmlogger/ol-node-01
directory.ls -l /var/log/pcp/pmlogger/ol-node-01/
Example Output:
[oracle@ol-node-01 ~]$ ls -l /var/log/pcp/pmlogger/ol-node-01/ total 47076 -rw-r--r--. 1 pcp pcp 36889492 May 8 19:12 20240508.18.31.0 -rw-r--r--. 1 pcp pcp 3492 May 8 19:12 20240508.18.31.index -rw-r--r--. 1 pcp pcp 666790 May 8 19:12 20240508.18.31.meta -rw-r--r--. 1 pcp pcp 277 May 8 18:31 Latest -rw-r--r--. 1 pcp pcp 28609 May 8 18:31 pmlogger.log
Note: In this example, the archive name is
20240508.18.31.0
. Substitute the actual archive name for your environment in the following steps. -
Use the
pmdumplog
command to view the host, timezone, and time period of the archive.pmdumplog -L /var/log/pcp/pmlogger/ol-node-01/<your-archive-name>
Example Output:
[oracle@ol-node-01 ~]$ pmdumplog -L /var/log/pcp/pmlogger/ol-node-01/<your-archive-name> Log Label (Log Format Version 2) Performance metrics from host ol-node-01 commencing Wed May 8 18:31:28.901169 2024 ending Wed May 8 19:17:58.936852 2024 Archive timezone: GMT PID for pmlogger: 133063
You can run almost any PCP command against the archive by including the
-a
option and providing the archive name as an argument. -
Run the
pcp
utility on the archive.sudo pcp -a /var/log/pcp/pmlogger/ol-node-01/<your-archive-name>
Example Output:
[oracle@ol-node-01 ~]$ sudo pcp -a /var/log/pcp/pmlogger/ol-node-01/<your-archive-name> Performance Co-Pilot configuration on ol-node-01: archive: /var/log/pcp/pmlogger/ol-node-01/<your-archive-name> platform: hardware: 4 cpus, 1 disk, 1 node, 31828MB RAM timezone: GMT pmcd: Version unknown,
-
List all performance metrics enabled at the time the archive was created.
pminfo -a /var/log/pcp/pmlogger/ol-node-01/<your-archive-name>
-
View disk write operations per partition from the archive. Pipe the command to
less
to show header information.pmval disk.partitions.write -a /var/log/pcp/pmlogger/ol-node-01/<your-archive-name> | less
-
View outgoing metrics from the ens3 network interface from the archive. Pipe the command to
less
to show header information.pmrep -i ens3 -v network.interface.out -a /var/log/pcp/pmlogger/ol-node-01/<your-archive-name> | less
Configure remote monitoring of a 2nd PCP collector host
In this section, you configure the ol-node-02 instance as a PCP collector. You perform additional steps to allow the ol-node-01 instance to monitor performance metrics on the remote host.
-
Open a new terminal and connect via SSH to the ol-node-02 instance.
ssh oracle@<ip_address_of_ol-node-02>
-
Install the
pcp
package on the ol-node-02 instance.sudo dnf install pcp
-
Enable and start the Performance Metrics Collector Daemon (
pmcd
) and the Performance Metrics Logger (pmlogger
) collector host services.sudo systemctl enable --now pmcd pmlogger
-
Enable the PCP client tools to connect to a remote PMCD by editing the
/etc/sysconfig/pmcd file
. ChangePMCD_LOCAL=1
toPMCD_LOCAL=0
. Save the file and exit thevi
editor.cd /etc/sysconfig sudo vi pmcd
-
Run the following command to restart the PMCD.
sudo /usr/libexec/pcp/lib/pmcd start
-
The PCP client tools communicate with PMCD over the network using TCP port 44321 by default. Use the
firewall-cmd
command to add port44321/tcp
.sudo firewall-cmd --add-port=44321/tcp
-
Return to the terminal for the ol-node-01 instance.
Note: All remaining tasks in this lab are performed from the ol-node-01 instance.
-
Use the
firewall-cmd
command to add port44321/tcp
.sudo firewall-cmd --add-port=44321/tcp
To enable centralized archive collection on the monitoring host, configure
pmlogger
to fetch performance metrics from collector hosts. Add the ol-node-02 instance to the pmlogger configuration file,/etc/pcp/pmlogger/control
and then restart thepmlogger
service on the monitoring host. -
Use the
vi
editor to edit/etc/pcp/pmlogger/control
.sudo vi /etc/pcp/pmlogger/control
-
While in
vi
, locate the following section:# remote host #remote n n PCP_ARCHIVE_DIR/remote -r -T24h10m -c config.remote
-
Uncomment the line beginning with
#remote
and replace each occurrence ofremote
in the line withol-node-02
as follows:ol-node-02 n n PCP_ARCHIVE_DIR/ol-node-02 -r -T24h10m -c config.ol-node-02
-
Save the file and exit the
vi
editor. -
Restart the
pmlogger
service.sudo systemctl restart pmlogger
-
Verify that you now have ol-node-01 and ol-node-02 directories in
/var/log/pcp/pmlogger
directory.ls -l /var/log/pcp/pmlogger
Example Output:
[oracle@ol-node-01 ~]$ ls -l /var/log/pcp/pmlogger total 12 drwxrwxr-x. 2 pcp pcp 4096 May 8 20:26 ol-node-01 drwxrwxr-x. 2 pcp pcp 119 May 8 20:26 ol-node-02 -rw-r--r--. 1 pcp pcp 131 May 8 20:26 pmlogger_check.log -rw-r--r--. 1 pcp pcp 315 May 8 18:31 pmlogger_check.log.prev
-
Verify that you can view PCP services on remote host ol-node-02.
sudo pcp -h ol-node-02
Example Output:
[oracle@ol-node-01 ~]$ sudo pcp -h ol-node-02 Performance Co-Pilot configuration on ol-node-02: platform: Linux ol-node-02 5.15.0-204.147.6.2.el9uek.x86_64 #2 SMP Thu Mar 7 22:09:50 PST 2024 x86_64 hardware: 4 cpus, 1 disk, 1 node, 31828MB RAM timezone: GMT services: pmcd pmcd: Version 6.2.0-2, 12 agents, 6 clients pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm jbd2 dm openmetrics pmlogger: primary logger: /var/log/pcp/pmlogger/ol-node-02/20240509.12.46 pmie: primary engine: /var/log/pcp/pmie/ol-node-02/pmie.log
-
Verify that you can view the PCP services from the local archive for host ol-node-02. View the archives in the
/var/log/pcp/pmlogger/ol-node-02
directory.ls -l /var/log/pcp/pmlogger/ol-node-02/
Example Output:
[oracle@ol-node-01 ~]$ ls -l /var/log/pcp/pmlogger/ol-node-02/ total 3500 -rw-r--r--. 1 pcp pcp 2700040 May 9 13:04 20240509.12.50.0 -rw-r--r--. 1 pcp pcp 492 May 9 13:04 20240509.12.50.index -rw-r--r--. 1 pcp pcp 374876 May 9 13:04 20240509.12.50.meta -rw-r--r--. 1 pcp pcp 277 May 9 12:50 Latest -rw-r--r--. 1 pcp pcp 34923 May 9 12:50 pmlogger.log
Note: In this example, the archive name is
20240509.12.50.0
. Substitute the actual archive name for your environment in the following command. -
View PCP services from the archive for remote host ol-node-02. Substitute the actual archive name for your environment in the following command.
sudo pcp -a /var/log/pcp/pmlogger/ol-node-02/<your-archive-name>
Example Output:
[oracle@ol-node-01 ~]$ sudo pcp -a /var/log/pcp/pmlogger/ol-node-02/<your-archive-name> Performance Co-Pilot configuration on ol-node-02: archive: /var/log/pcp/pmlogger/ol-node-02/<your-archive-name> platform: Linux ol-node-02 5.15.0-204.147.6.2.el9uek.x86_64 #2 SMP Thu Mar 7 22:09:50 PST 2024 x86_64 hardware: 4 cpus, 1 disk, 1 node, 31828MB RAM timezone: GMT services: pmcd pmcd: Version 6.2.0-2, 12 agents, 4 clients pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm jbd2 dm openmetrics pmie: primary engine: /var/log/pcp/pmie/ol-node-02/pmie.log
You can run almost any PCP command against the remote host by including the
-h
option and providing the host name as an argument. -
Run the
pmstat
command for remote host ol-node-02.pmstat -h ol-node-02
Example Output:
[oracle@ol-node-01 ~]$ pmstat -h ol-node-02 @ Thu May 9 13:19:35 2024 loadavg memory swap io system cpu 1 min swpd free buff cache pi po bi bo in cs us sy id 0.00 0 29667m 4204 1539m 0 0 0 17 114 170 0 0 100 0.00 0 29666m 4204 1540m 0 0 0 4 117 164 0 0 100 ^C
-
Run the
pmval
command to view live disk write operations per partition for remote host ol-node-02.pmval disk.partitions.write -h ol-node-02
Example Output:
[oracle@ol-node-01 ~]$ pmval disk.partitions.write -h ol-node-02 metric: disk.partitions.write host: ol-node-02 semantics: cumulative counter (converting to rate) units: count (converting to count / sec) samples: all sda1 sda2 sda3 0.0 0.0 1.999 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ^C
Monitor performance with pmchart GUI
pmchart
is a graphical utility provided by the pcp-gui
package. pmchart
allows you to plot PCP performance metrics from a live host or from PCP archives. Multiple charts can be displayed simultaneously and many aspects of pmchart
can be customized through the interface.
This section of the lab assumes you have two (2) terminal windows open, and that you are logged into the ol-node-01 instance from one terminal window, and that you are logged into the ol-node-02 instance from the second terminal windows.
-
To begin this lab, logoff from both Oracle Linux instances so that you are at the
[luna.user@lunabox Desktop]$
prompt in both terminal windows.exit
-
From one terminal, access the
vnc
session on the ol-node-01 instance using a SSH tunnel.ssh -L 5910:localhost:5901 oracle@<ip_address_of_ol-node-01>
-
From the 2nd terminal, open vncviewer. Provide the password of
oracle
when prompted.vncviewer localhost:10
-
The Oracle Linux desktop appears on the ol-node-01 instance. Click on Activities and open a Terminal window.
-
Enter the command
pmchart
to open the graphical interface.pmchart
-
From the File menu select the ‘New Chart’ option.
The New Chart dialog box appears. The three tabs on the right side of the window provide the following settings:
- Chart: Properties related to the entire chart (Title, Legend settings, Y-Axis scaling, etc.)
- Metrics: Available metrics (performance data) for plotting in the chart
- Plots: Properties related to each individual chart plot (color, label)
-
From the Metrics tab, select a performance metric, for example
kernel.cpu.util.sys
, by drilling down as shown. -
Click OK and after a few seconds, data will begin to appear on the chart.
Click the ‘Zoom In’ button to change the scale for better viewing.
-
Add the ol-node-02 instance by clicking ‘New Chart’, then click the ‘Add a host’ button on the New Chart dialog box. Enter ol-node-02 as the Hostname and click OK.
-
You can now select available performance metrics from ol-node-02. You can scroll down to view available metrics for ol-node-02, or you can double-click the ol-node-01 entry from the Metrics tab to collapse the metrics for ol-node-01 and view metrics for ol-node-02.
-
From the Metrics tab, select
kernel.cpu.util.sys
for ol-node-02 and click OK. You now have a 2nd chart showing the same performance metrics for each host.
Summary
The Performance Co-Pilot (PCP) is a toolkit designed for monitoring and managing system-level performance. Oracle Linux hosts can be configured as PCP collectors, those that collect and extract performance data from various sources, and can be configured as PCP monitors, those that display data collected from hosts or archives that have the PCP collector installed.
For More Information
Oracle Linux Performance Co-Pilot Documentation
More Learning Resources
Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.
For product documentation, visit Oracle Help Center.
Work With Performance Co-Pilot on Oracle Linux
F98303-01
May 2024