Go to main content

Managing System Information, Processes, and Performance in Oracle® Solaris 11.4

Exit Print View

Updated: October 2019
 
 

Monitoring System Activities With the sar Command

    You can use the sar command to perform the following tasks:

  • Organize and view data about system activity.

  • Access system activity data on a special request basis.

  • Generate automatic reports to measure and monitor system performance as well as special request reports to pinpoint specific performance problems. For information about how to set up the sar command to run on your system as well as a description of these tools, see Collecting System Activity Data Automatically.

For a detailed description of this command, see the sar(1) man page.

Monitoring File Access

You can display file access operation statistics with the sar -a command.

The larger the reported values for these operating system routines, the more time the kernel is spending to access user files. The amount of time reflects how heavily programs and applications are using the file systems. The –a option is helpful to view whether an application is disk-dependent.

The following list describes the file access routines that are displayed by the –a option.

iget/s

The number of requests made for inodes that were not in the directory name look-up cache (DNLC).

namei/s

The number of file system path searches per second. If namei does not find a directory name in the DNLC, it calls iget to get the inode for either a file or directory. Hence, most igets are the result of DNLC misses.

dirbk/s

The number of directory block reads issued per second.

Example 20  Reporting File Access
$ sar -a
SunOS example-server 5.11 11.4.0.15.0 sun4v    09/07/2019

00:00:00  iget/s namei/s dirbk/s
01:00:01       0     112       0
02:00:00       0     136       0
...
09:20:00       0     156       0
09:40:01       0      96       0
10:00:00       0      83       0
10:20:00       0     119       0

Average        0     115       0

Monitoring Buffer Activity

You can display buffer activity statistics with the sar -b command. The buffer is used to cache metadata. Metadata includes inodes, cylinder group blocks, and indirect blocks.

The following list describes the buffer activities that are displayed by the –b option.

bread/s

Average number of reads per second that are submitted to the buffer cache from the disk

lread/s

Average number of logical reads per second from the buffer cache

%rcache

Fraction of logical reads that are found in the buffer cache (100 % minus the ratio of bread/s to lread/s)

bwrit/s

Average number of physical blocks (512 bytes) that are written from the buffer cache to disk per second

lwrit/s

Average number of logical writes to the buffer cache per second

%wcache

Fraction of logical writes that are found in the buffer cache (100 % minus the ratio of bwrit/s to lwrit/s)

pread/s

Average number of physical reads per second that use character device interfaces

pwrit/s

Average number of physical write requests per second that use character device interfaces

The most important entries are the cache hit ratios %rcache and %wcache. These entries measure the effectiveness of system buffering. If %rcache falls below 90 percent or if %wcache falls below 65 percent, you might be able to improve performance by increasing the buffer space.

Example 21  Reporting Buffer Activity

The following example of sar -b command output shows that the %rcache and %wcache buffers are not causing any slowdowns. All the data is within acceptable limits.

$ sar -b
SunOS example-server 5.11 11.4.0.15.0 sun4v    09/07/2019

00:00:00 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
01:00:01       0       0     100       0       0     100       0       0
02:00:00       0       0     100       0       0     100       0       0
03:00:00       0       0     100       0       0     100       0       0

Average        0       0     100       0       0     100       0       0

Monitoring System Call Statistics

You can display system call statistics by using the sar -c command. Typically, reads and writes account for about half of the total system calls. However, the percentage varies greatly with the activities that are being performed by the system.

The following list describes the system call categories that are reported by the –c option.

scall/s

The number of all types of system calls per second, which is generally about 30 per second on a system with four to six users.

sread/s

The number of read system calls per second.

swrit/s

The number of write system calls per second.

fork/s

The number of fork system calls per second, which is about 0.5 per second on a system with four to six users. This number increases if shell scripts are running.

exec/s

The number of exec system calls per second. If exec/s divided by fork/s is greater than 3, look for inefficient PATH variables.

rchar/s

The number of characters (bytes) transferred by read system calls per second.

wchar/s

The number of characters (bytes) transferred by write system calls per second.

Example 22  Reporting System Call Statistics

The following example shows output from the sar -c command.

$ sar -c
SunOS example-server 5.11 11.4.0.15.0 sun4v    09/07/2019

00:00:00 scall/s sread/s swrit/s  fork/s  exec/s rchar/s wchar/s
01:00:01    1276      24       3    0.68    0.71  170648  123527
02:00:00    1390      30       4    0.97    1.03  202074  136693
03:00:00    1272      25       4    0.66    0.69  196961  152584
04:00:00    1267      24       3    0.67    0.69  207036  161820
05:00:00    1315      26       4    0.73    0.77  232693  175446
06:00:01    1290      25       4    0.67    0.70  234286  188378
07:00:00    1306      26       4    0.76    0.80  195116  143241
08:00:00    1265      25       3    0.66    0.69  130554   83942
08:20:00    1626      44       9    1.21    1.27  175683   94947

Average     1297      26       4    0.71    0.75  187263  137889

Monitoring Disk Activity

You can display disk activity statistics with the sar -d command.

The following list describes the system call categories that the –d option reports.

device

Name of the disk device that is being monitored.

%busy

Portion of time the device was busy servicing a transfer request.

avque

Average number of requests during the time the device was busy servicing a transfer request.

r+w/s

Number of read-and-write transfers to the device, per second.

blks/s

Number of 512-byte blocks that are transferred to the device, per second.

avwait

Average time, in milliseconds, that transfer requests wait in the queue. This time is measured only when the queue is occupied.

avserv

Average time, in milliseconds, for a transfer request to be completed by the device. For disks, this value includes seek times, rotational latency times, and data transfer times.

Example 23  Reporting Disk Activity

The following example shows output from the sar -d command.

$ sar -d
SunOS example-server 5.11 11.4.0.15.0 sun4v    09/07/2019

00:00:00   device         %busy   avque   r+w/s  blks/s  avwait  avserv

01:00:01   iscsi_se          0     0.1       0       1  4432.8     1.2
           sd0.t1.i          0     0.0       0       1     0.0     1.3
           nfs4              0     0.0       0       0     0.8     0.9
           nfs11             0     0.0       0       0     0.0     0.0
           nfs1694           0     0.0       0       0     0.0     0.0
           nfs2981           0     0.0       0       0     0.0     0.0
           nfs5471           0     0.0       0       0     0.0     0.0
           nfs20629          0     0.0       0       0     0.0     0.6
           nfs20631          0     0.0       0       0     0.0     0.7
           sd0               0     0.0       0       1     0.0     1.3
           sd0,a             0     0.0       0       1     0.0     1.3
           sd0,h             0     0.0       0       0     0.0     0.0
           vdc0              2     0.1       8     529     0.0    11.4

02:00:00   iscsi_se          0     0.1       0       0     0.0     0.0
           sd0.t1.i          0     0.0       0       0     0.0     0.0
           nfs4              0     0.0       0       0     0.8     0.7
...
Average    iscsi_se          0     0.1       0       2  1479.7     3.9
           sd0.t1.i          0     0.0       0       2     0.0     4.0
           nfs4              0     0.0       0       0     0.0     0.6
...

Note that queue lengths and wait times are measured when something is in the queue. If %busy is small, large queues and service times probably represent the periodic efforts by the system to ensure that altered blocks are promptly written to the disk.

Monitoring Page-Out and Memory

You can use the sar -g command to display page-out and memory freeing acCtivities in averages.

The output displayed by the sar -g command is a good indicator memory needs. Use the ps -elf command to show the number of cycles that are used by the page daemon. A high number of cycles, combined with high values for the pgfree/s and pgscan/s fields, indicates a memory shortage.

The sar -g command also shows whether inodes are being recycled too quickly and causing a loss of reusable pages.

The following list describes the output from the –g option.

pgout/s

The number of page-out requests per second.

ppgout/s

The actual number of pages that are paged-out per second. A single page-out request might involve paging-out multiple pages.

pgfree/s

The number of pages per second that are placed on the free list.

pgscan/s

The number of pages per second that are scanned by the page daemon. If this value is high, the page daemon is spending a lot of time checking for free memory. This situation implies that more memory is needed.

%ufs_ipf

The percentage of ufs inodes taken off the free list by iget that had reusable pages associated with them. These pages are flushed and cannot be reclaimed by processes. Thus, this field represents the percentage of igets with page flushes. A high value indicates that the free list of inodes is page-bound, and that the number of ufs inodes need to be increased.

Example 24  Reporting Page-Out and Memory

The following example shows output from the sar -g command.

$ sar -g
SunOS example-server 5.11 11.4.0.15.0 sun4v    09/07/2019

00:00:00  pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf
01:00:01     0.00     0.00     0.00     0.00     0.00
02:00:00     0.00     0.00     0.00     0.00     0.00
03:00:00     0.00     0.00     0.00     0.00     0.00
04:00:00     0.00     0.00     0.00     0.00     0.00
05:00:00     0.00     0.00     0.00     0.00     0.00
06:00:01     0.00     0.00     0.00     0.00     0.00
07:00:00     0.00     0.00     0.00     0.00     0.00
08:00:00     0.00     0.00     0.00     0.00     0.00
08:20:00     0.00     0.00     0.00     0.00     0.00
08:40:00     0.00     0.00     0.00     0.00     0.00
09:00:00     0.00     0.00     0.00     0.00     0.00
09:20:00     0.00     0.00     0.00     0.00     0.00
09:40:01     0.00     0.00     0.00     0.00     0.00
10:00:00     0.00     0.00     0.00     0.00     0.00
10:20:00     0.00     0.00     0.00     0.00     0.00

Average      0.00     0.00     0.00     0.00     0.00

Monitoring Kernel Memory Allocation

The Kernel Memory Allocation (KMA) allows a kernel subsystem to allocate and free memory as needed.

    Rather than statically allocating the maximum amount of memory that might be needed under peak load, the KMA divides requests for memory into three categories:

  • Small (less than 256 bytes)

  • Large (512 bytes to 4 Kbytes)

  • Oversized (greater than 4 Kbytes)

The KMA keeps two pools of memory to satisfy small requests and large requests. The oversized requests are satisfied by allocating memory from the system page allocator.

You can use the sar -k command to report on activities of the Kernel Memory Allocator (KMA). The sar -k command is useful if you are checking a system that is being used to write drivers or STREAMS that use KMA resources. Any driver or module that uses KMA resources but does not specifically return the resources before it exits, can create a memory leak. A memory leak causes the amount of memory that is allocated by KMA to increase over time. Thus, if the alloc fields of the sar -k command increase steadily over time, there might be a memory leak. Another indication of a memory leak is failed requests. If this problem occurs, a memory leak has probably caused KMA to be unable to reserve and allocate memory.

If it appears that a memory leak has occurred, you should check any drivers or STREAMS that might have requested memory from KMA and not returned it.

The following list describes the output from the –k option.

sml_mem

The amount of memory in bytes that the KMA has available in the small memory request pool. In this pool, a small request is less than 256 bytes.

alloc

The amount of memory in bytes that the KMA has allocated from its small memory request pool to small memory requests.

fail

The number of requests for small amounts of memory that failed.

lg_mem

The amount of memory in bytes that the KMA has available in the large memory request pool. In this pool, a large request is from 512 bytes to 4 Kbytes.

alloc

The amount of memory in bytes that the KMA has allocated from its large memory request pool to large memory requests.

fail

The number of failed requests for large amounts of memory.

ovsz_alloc

The amount of memory that is allocated for oversized requests, which are requests that are greater than 4 Kbytes. These requests are satisfied by the page allocator. Thus, there is no pool.

fail

The number of failed requests for oversized amounts of memory.

Example 25  Reporting Kernel Memory Allocation

The following example shows an abbreviated output from the sar -k command.

$ sar -k
SunOS example-server 5.11 11.4.0.15.0 sun4v    09/07/2019

00:00:00 sml_mem   alloc  fail  lg_mem   alloc  fail  ovsz_alloc  fail
01:00:01 2334107392 2634568704 1860748 2826780672 3027058432 31666  1714692096     0
02:00:00 2334107392 2639957504 1860748 3328139264 3561145088 31666  1714692096     0
03:00:00 2334148352 2644719616 1860748 3881467904 4150487808 31666  1714692096     0
04:00:00 2334213888 2650413568 1860748 185196544 489396352 31666  1714692096     0
05:00:00 2334287616 2655426560 1860748 826310656 1153170304 31666  1714692096     0
06:00:01 2334369536 2660379136 1860748 1518903296 1869017728 31666  1714692096     0
07:00:00 2334426880 2665492480 1860748 2046361600 2417197056 31666  1714692096     0
08:00:00 2334459648 2669230080 1860748 2361794560 2749543168 31666  1714692096     0
08:20:00 2334476032 2671023616 1860748 2479095808 2869149184 31666  1714692096     0
08:40:00 2334492416 2672560640 1860748 2603687936 3001746688 31666  1714692096     0
09:00:00 2334500608 2673795072 1860748 2733891584 3140963072 31666  1714692096     0
09:20:00 2334516992 2675175680 1860748 2865938432 3275326208 31666  1714692096     0
09:40:01 2334533376 2677078016 1860748 3005399040 3428043776 31666  1714692096     0
10:00:00 2334549760 2678332160 1860748 3146645504 3579805952 31666  1714692096     0
10:20:00 2334566144 2679967232 1860748 3301113856 3741610240 31666  1714692096     0

Average  2334383616 2663207936 1860748 2474048512 2830244096 31666  1714692096     0

Monitoring Interprocess Communication

You can use the sar -m command to report interprocess communication activities.

These figures are usually zero (0.00), unless you are running applications that use messages or semaphores.

The following list describes the output from the –m option.

msg/s

The number of message operations (send and receive) per second

sema/s

The number of semaphore operations per second

The following abbreviated example shows output from the sar -m command.

$ sar -m
SunOS example-server 5.11 11.4.0.15.0 sun4v    09/07/2019

00:00:00   msg/s  sema/s
01:00:01    0.00    0.00
02:00:00    0.00    0.00
...
10:20:00    0.00    0.00

Average     0.00    0.00

Monitoring Page-In Activity

You can use the sar -p command to report page-in activity, which includes protection and translation faults.

The following list describes the reported statistics from the –p option.

atch/s

The number of page faults per second that are satisfied by reclaiming a page currently in memory (attaches per second). Instances include reclaiming an invalid page from the free list and sharing a page of text that is currently being used by another process. An example is two or more processes that are accessing the same program text.

pgin/s

The number of times per second that file systems receive page-in requests.

ppgin/s

The number of pages paged in per second. A single page-in request, such as a soft-lock request (see slock/s) or a large block size, might involve paging-in multiple pages.

pflt/s

The number of page faults from protection errors. Instances of protection faults indicate illegal access to a page and copy-on-writes. Generally, this number consists primarily of copy-on-writes.

vflt/s

The number of address translation page faults per second. These faults are known as validity faults, faults occur when a valid process table entry does not exist for a given virtual address.

slock/s

The number of faults per second caused by software lock requests that require physical I/O. An example of the occurrence of a soft-lock request is the transfer of data from a disk to memory. The system locks the page that is to receive the data so that the page cannot be claimed and used by another process.

Example 26  Reporting Page-In Activity

The following example shows output from the sar -p command.

$ sar -p
SunOS example-server 5.11 11.4.0.15.0 sun4v    09/07/2019

00:00:00  atch/s  pgin/s ppgin/s  pflt/s  vflt/s slock/s
01:00:01   56.31    0.00    0.00  167.66  299.39    0.02
02:00:00   66.17    0.01    0.03  218.07  349.68    0.03
03:00:00   56.92    0.00    0.01  156.83  299.16    0.02
04:00:00   56.49    0.00    0.00  157.43  300.03    0.02
05:00:00   59.62    0.21    0.85  171.29  313.03    0.02
06:00:01   57.71    0.03    0.05  150.49  303.41    0.02
07:00:00   60.85    0.14    0.21  177.63  306.82    0.02
08:00:00   57.45    0.00    0.00  156.70  274.23    0.02
08:20:00   77.47    0.01    0.02  312.96  391.30    0.04
08:40:00   46.90    0.01    0.01   93.34  226.69    0.01
09:00:00   43.93    0.00    0.00   67.28  209.63    0.00
09:20:00   75.66    0.00    0.00  321.04  390.62    0.04
09:40:01   46.68    0.01    0.01   93.83  229.32    0.02
10:00:00   44.07    0.00    0.01   67.41  219.60    0.00
10:20:00   59.93    0.00    0.00  161.27  303.64    0.04

Average    58.36    0.04    0.11  167.27  300.26    0.02

Monitoring Queue Activity

    You can use the sar –q command to report the following information:

  • The average queue length while the queue is occupied.

  • The percentage of time that the queue is occupied.

The following list describes the queue information from the –q option.

runq-sz

The number of kernel threads in memory that are waiting for a CPU to run. Typically, this value should be less than 2. Consistently higher values mean that the system might be CPU-bound.

%runocc

The percentage of time that the dispatch queues are occupied.

swpq-sz

The average number of swapped out processes.

%swpocc

The percentage of time in which the processes are swapped out.

Example 27  Reporting Queue Activity

The following example shows output from the sar -q command. If the %runocc value is high (greater than 90 percent) and the runq-sz value is greater than 2, the CPU is heavily loaded and response is degraded. In this case, additional CPU capacity might be required to obtain acceptable system response.

# sar -q
SunOS example-server 5.11 11.4.0.15.0 sun4v    09/07/2019

00:00:00 runq-sz %runocc swpq-sz %swpocc
01:00:01     1.2       7     0.0       0
02:00:00     1.2       5     0.0       0
03:00:00     1.2       6     0.0       0
04:00:00     1.2       6     0.0       0
05:00:00     1.2       5     0.0       0
06:00:01     1.2       4     0.0       0
07:00:00     1.2       3     0.0       0
08:00:00     1.2       8     0.0       0
08:20:00     1.2       6     0.0       0
08:40:00     1.2       4     0.0       0

Average      1.2       5     0.0       0

Monitoring Unused Memory

You can use the sar -r command to report the number of memory pages and swap-file disk blocks that are currently unused.

The following list describes the output from the –r option.

freemem

The average number of memory pages that are available to user processes over the intervals sampled by the command. Page size is system-dependent.

freeswap

The number of 512-byte disk blocks that are available for page swapping.

Example 28  Reporting Unused Memory

The following example shows output from the sar -r command.

$ sar -r
SunOS example-server 5.11 11.4.0.15.0 sun4v    09/07/2019

00:00:00 freemem freeswap
01:00:01 2033446 143413667
02:00:00 1976436 142455652
03:00:00 1913957 141472967
04:00:00 1843041 140330892
05:00:00 1767369 138958248
06:00:01 1684910 137870401
07:00:00 1602032 136537114
08:00:00 1558301 135857941
08:20:00 1530449 135420182
08:40:00 1514288 135150442
09:00:00 1497581 134858710
09:20:00 1481513 134649032
09:40:01 1465218 134331511
10:00:00 1446124 134037258
10:20:00 1426441 133654337

Average  1725814 138477241

Monitoring CPU Utilization

You can use the sar -u command to display CPU utilization statistics.

The sar command without any options is equivalent to the sar -u command. At any given moment, the processor is either busy or idle. When busy, the processor is in either user mode or system mode. When idle, the processor is either waiting for I/O completion or has no work to do.

The following list describes the CPU utilization statistics from the –u option.

%usr

The percentage of time that the processor is in user mode

%sys

The percentage of time that the processor is in system mode

%wio

The percentage of time that the processor is idle and waiting for I/O completion

%idle

The percentage of time that the processor is idle and not waiting for I/O

A high %wio value generally means that a disk slowdown has occurred.

Example 29  Reporting CPU Utilization

The following example shows output from the sar -u command.

$ sar -u
00:00:04    %usr    %sys    %wio   %idle
01:00:00       0       0       0     100
02:00:01       0       0       0     100
03:00:00       0       0       0     100
04:00:00       0       0       0     100
05:00:00       0       0       0     100
06:00:00       0       0       0     100
07:00:00       0       0       0     100
08:00:00       0       0       0     100
08:20:00       0       0       0      99
08:40:01       0       0       0      99
09:00:00       0       0       0      99
09:20:00       0       0       0      99
09:40:00       4       1       0      95
10:00:00       4       2       0      94
10:20:00       1       1       0      98
10:40:00      18       3       0      79
11:00:00      25       3       0      72

Average        2       0       0      98

Monitoring System Table Status

You can use the sar -v command to report the status of the process table, inode table, file table, and shared memory record table.

The following list describes the system table statuses from the –v option.

proc-sz

The number of process entries (proc structures) that are currently being used, or allocated, in the kernel.

inod-sz

The total number of inodes in memory compared to the maximum number of inodes that are allocated in the kernel. This number is not a strict high watermark. The number can overflow.

file-sz

The size of the open system file table. sz is given as 0, because space is allocated dynamically for the file table.

ov

The overflows that occur between sampling points for each table.

lock-sz

The number of shared memory record table entries that are currently being used, or allocated, in the kernel. sz is given as 0 because space is allocated dynamically for the shared memory record table.

Example 30  Reporting System Table Status

The following abbreviated example shows output from the sar -v command. This example shows that all tables are large enough to have no overflows. These tables are all dynamically allocated based on the amount of physical memory.

$ sar -v
00:00:04  proc-sz    ov  inod-sz    ov  file-sz    ov   lock-sz
01:00:00   69/8010    0 3476/34703    0    0/0       0    0/0   
02:00:01   69/8010    0 3476/34703    0    0/0       0    0/0   
03:00:00   69/8010    0 3476/34703    0    0/0       0    0/0   
04:00:00   69/8010    0 3494/34703    0    0/0       0    0/0 

Monitoring Swapping Activity

You can use the sar -w command to report swapping and switching activity.

The following list describes the output from the –w option.

swpin/s

The number of LWP transfers into memory per second.

bswin/s

The number of blocks transferred for swap-ins per second. /* (float)PGTOBLK(xx->cvmi.pgswapin) / sec_diff */.

swpot/s

The average number of processes that are swapped out of memory per second. If the number is greater than 1, you might need to increase memory.

bswot/s

The number of blocks that are transferred for swap-outs per second.

pswch/s

The number of kernel thread switches per second.


Note -  All process swap-ins include process initialization.
Example 31  Reporting Swap Activity

The following example shows output from the sar -w command.

$ sar -w
00:00:04 swpin/s bswin/s swpot/s bswot/s pswch/s
01:00:00    0.00     0.0    0.00     0.0     132
02:00:01    0.00     0.0    0.00     0.0     133
...
09:20:00    0.00     0.0    0.00     0.0     132
09:40:00    0.00     0.0    0.00     0.0     335
10:00:00    0.00     0.0    0.00     0.0     601
10:20:00    0.00     0.0    0.00     0.0     353
10:40:00    0.00     0.0    0.00     0.0     747
11:00:00    0.00     0.0    0.00     0.0     804

Average     0.00     0.0    0.00     0.0     198

Monitoring Terminal Activity

You can use the sar -y command to monitor terminal device activities.If you have a lot of terminal I/O, you can use this report to determine whether any bad lines exist.

The following list describes the output from the –y option.

rawch/s

Input characters (raw queue) per second

canch/s

Input characters that are processed by canon (canonical queue) per second

outch/s

Output characters (output queue) per second

rcvin/s

Receiver hardware interrupts per second

xmtin/s

Transmitter hardware interrupts per second

mdmin/s

Modem interrupts per second

The number of modem interrupts per second (mdmin/s) should be close to zero. The receive and transmit interrupts per second (xmtin/s and rcvin/s) should be less than or equal to the number of incoming or outgoing characters, respectively. If not, check for bad lines.

Example 32  Reporting Terminal Activity

The following example shows output from the sar -y command.

$ sar -y
00:00:04 rawch/s canch/s outch/s rcvin/s xmtin/s mdmin/s
01:00:00       0       0       0       0       0       0
...
09:40:00       0       0       1       0       0       0
10:00:00       0       0      37       0       0       0
10:20:00       0       0       0       0       0       0
10:40:00       0       0       3       0       0       0

Average        0       0       1       0       0       0

Monitoring Overall System Performance

You can use the sar -A command to display statistics from all options to provide a view of overall system performance.

This command provides a more global perspective. If data from more than a single time segment is shown, the report includes averages.