Using sysstat to Monitor and Review System Resource Usage Statistics

Enable the sysstat service to collect system logs, then use the sar, mpstat, top, iostat, and iotop commands to review detailed information about CPU load, memory use, and remaining storage capacity.

Before running the sar, mpstat, or iostat commands, enable the sysstat service and its related systemd timers so that it generates log entries every 10 minutes while the system is running, and then retains that information for every day of the current month:

sudo systemctl enable --now sysstat sysstat-collect.timer sysstat-summary.timer

For more information about running and configuring system services, see Oracle Linux 8: Managing the System With systemd.

The information that the sysstat service collects is stored in directory paths following the /var/log/sa/saDD pattern. To display the contents of the sar log of a specific day of the current month, run the following command:

sudo sar -A -f /var/log/sa/saDD

You can also use the sar command to create a customized log that contains a record of specific information that you want to monitor. Use the following syntax:

sudo sar -o datafile seconds count >/dev/null 2>&1 &

In the previous command, datafile is the full path to the customized log where you want to store the information, while count represents the number of samples to record. With this command, the sar process runs in the background and collects the data.

To display the results of this monitoring, run the following command, specifying the log file:

sudo sar -A -f datafile

Reviewing CPU Usage Statistics

To review information about the system CPU load average, run the following command:

sar -q

14:57:55     LINUX RESTART	(2 CPU)

03:00:01 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
03:10:01 PM         0       214      0.19      0.30      0.22         0
03:20:01 PM         0       212      0.00      0.05      0.10         0
...
Average:      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
Average:            1       212      0.12      0.08      0.05         0

The previous command provides more detailed information than the uptime command:

uptime

21:25:34 up  6:28,  1 user,  load average: 0.02, 0.10, 0.04

To review CPU usage statistics for each CPU core and averaged the data across all the CPU cores, run the following command:

sar -u -P ALL

03:00:01 PM  CPU  %user   %nice   %system   %iowait  %steal   %idle
03:10:01 PM  all  8.30    0.00    2.20      0.22     0.10     89.18
03:10:01 PM    0  8.22    0.00    2.64      0.31     0.09     88.74
03:10:01 PM    1  8.39    0.00    1.77      0.13     0.10     89.61

The same information that was collected by the sysstat service can also be used with the mpstat command to view further detail that was collected about CPU resource usage:

mpstat -P ALL

05:10:01 PM  CPU    %usr   %nice    %sys   %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
05:10:01 PM  all    0.76    0.02    0.70      0.02    0.08    0.05    0.06    0.00    0.00   98.32
05:10:01 PM    0    0.75    0.01    0.68      0.00    0.09    0.03    0.07    0.00    0.00   98.37
05:10:01 PM    1    0.76    0.03    0.72      0.03    0.06    0.06    0.06    0.00    0.00   98.27

For a live summary of CPU usage for each running process, and a navigable interface for ending those processes, use the top command:

top

top - 22:13:34 up  7:16,  1 user,  load average: 0.00, 0.02, 0.01
Tasks: 149 total,   1 running, 148 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.2 hi,  0.0 si,  0.0 st
MiB Mem :  14705.5 total,  11738.9 free,    753.2 used,   2213.4 buff/cache
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.  13588.9 avail Mem
                    
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
34781 root      20   0   11384   7100   1700 S   1.0   0.0   0:00.03 pidstat
 2014 root      20   0   26216   3568   3056 S   0.7   0.0   0:04.78 OSWatcher
 1481 root      20   0  202628  38328  36792 S   0.3   0.3   0:02.94 sssd_nss
...

By default, the top command output lists the most CPU-intensive processes on the system. The upper section displays general information including the load averages over the past 1, 5, and 15 minutes, the number of running and sleeping processes or tasks, and total CPU, and memory usage.

The second table displays a list of processes, including the process ID number (PID), the process owner, CPU usage, memory usage, running time, and the command name. By default, the list is sorted by CPU usage, with the top consumer of CPU listed first.

To exit the top command, press the Ctrl + C keys in combination.

Reviewing Memory Usage Statistics

To review memory usage statistics, such as free (kbmemfree), available (kbavail), and used (kbmemused) memory, run the following command:

sar -r

The output also includes %memused, which is the percentage of physical memory in use.

To review memory paging statistics, such as page in (pgpgin/s) and page out (pgpgout/s), page faults and major faults, run the following command:

sar -B

The output also includes pgscank/s, which is the number of memory pages scanned by the kswapd daemon each second, and pgscand/s, which is the number of memory pages scanned directly each second.

To review swapping statistics, including pswpin/s and pswpout/s, which are the numbers of pages each second swapped in and out each second, run the following command:

sar -W

For a full list of options, run the sar --help command.

If %memused is near 100% and the scan rate is continuously over 200 pages each second, the system has a memory shortage.

When a system runs out of real or physical memory and starts using swap space, system performance deteriorates. If swap space is full then some programs or even the entire OS can malfunction. If the free or top commands indicate that little swap space remains available, then the system is running low on memory.

Output from the dmesg command might also include notifications about problems with physical memory that were detected at boot time.

Reviewing Storage Usage Statistics

To review the number of unused cache entries in the directory cache (dentunusd) and the numbers of in-use file handles (file-nr), inode handlers (inode-nr), and pseudo terminals (pty-nr), run the following command:

sar -v

12:00:01 AM dentunusd   file-nr  inode-nr    pty-nr
12:10:33 AM     80101      2944     73074         0
12:20:33 AM     79788      2944     72654         0

The iostat command monitors the loading of block I/O devices by observing the time that the devices are active relative to the average data transfer rates. You can use this information to adjust the system configuration to balance the I/O loading across disks and host adapters. The following is a sample of the command output:

iostat

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.69    0.05    0.77    0.00    0.03   98.46

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               1.05         2.32        25.19     611410    6640659
dm-0              0.62         2.07        20.22     545716    5329660
dm-1              0.70         0.02         4.97       4632    1308788

iostat -x reports extended statistics about block I/O activity at one second intervals, including %util, which is the percentage of CPU time spent handling I/O requests to a device, and avgqu-sz, which is the average queue length of I/O requests that were issued to that device. If %util approaches 100% or avgqu-sz is greater than 1, device saturation is occurring.

Use the sar -d command to review similar information on block I/O activity, including values for %util and avgqu-sz.

If the iotop package is installed, the iotop utility can be used to identify which processes are responsible for excessive disk I/O. iotop has a similar user interface to the top command. In its upper section, iotop displays the total disk input and output usage in bytes per second. In its earlier section, iotop displays I/O information for each process, including disk input output usage in bytes per second, the percentage of time spent swapping in pages from disk or waiting on I/O, and the command name. The following is a sample command output:

sudo iotop

Total DISK READ :	0.00 B/s | Total DISK WRITE :       0.00 B/s
Actual DISK READ:	0.00 B/s | Actual DISK WRITE:       0.00 B/s
    TID  PRIO  USER     DISK READ DISK WRITE>    COMMAND
      1 be/4 root        0.00 B/s    0.00 B/s systemd --switched-root --system --deserialize 16
      2 be/4 root        0.00 B/s    0.00 B/s [kthreadd]
...

While reviewing the output, use the arrow keys to change the sort field, and press A to switch the I/O units between bytes each second and total number of bytes, or O to switch between displaying all processes or only those processes that are performing I/O.

To exit the iotop command, press the Ctrl + C keys in combination.

The nfsiostat command reports I/O statistics for each NFS file system that's mounted. If this command isn't available, install the nfs-utils package.