This chapter describes procedures for monitoring system performance by using the vmstat, iostat, df, and sar commands. This is a list of the step-by-step instructions in this chapter.
The vmstat command now includes all system interrupts in the faults-in column.
In previous Solaris releases, this column did not report clock device interrupts.
For more information, see vmstat(1M).
You can use the vmstat command to report virtual memory statistics and information about system events as CPU load, paging, number of context switches, device interrupts, and system calls. The vmstat command can also display statistics on swapping, cache flushing, and interrupts.
The following table describes the fields in the vmstat output.
Table 24–1 Output From the vmstat Command
Category |
Field Name |
Description |
---|---|---|
procs |
|
Reports on the following: |
|
r |
The number of kernel threads in the dispatch queue |
|
b |
The number of blocked kernel threads that are waiting for resources |
|
w |
The number of swapped out LWPs that are waiting for processing resources to finish |
memory |
|
Reports on usage of real memory and virtual memory: |
|
swap |
Available swap space |
|
free |
Size of the free list |
page |
|
Reports on page faults and paging activity, in units per second: |
|
re |
Pages reclaimed |
|
mf |
Minor and major faults |
|
pi |
Kbytes paged in |
|
po |
Kbytes paged out |
|
fr |
Kbytes freed |
|
de |
Anticipated memory that is needed by recently swapped-in processes |
|
sr |
Pages scanned by the page daemon not currently in use. If sr does not equal zero, the page daemon has been running. |
disk |
|
Reports the number of disk operations per second, showing data on up to four disks |
faults |
|
Reports the trap/interrupt rates per second: |
|
in |
Interrupts per second |
|
sy |
System calls per second |
|
cs |
CPU context switch rate |
cpu |
|
Reports on the use of CPU time: |
|
us |
User time |
|
sy |
System time |
|
id |
Idle time |
For a more detailed description of this command, see vmstat(1M).
Collect virtual memory statistics by using the vmstat command with a time interval in seconds.
$ vmstat n |
n is the interval in seconds between reports.
The following example shows the vmstat display of statistics gathered at five-second intervals.
$ vmstat 5 procs memory page disk faults cpu r b w swap free re mf pi po fr de sr f0 s3 -- -- in sy cs us sy id 0 0 8 28312 668 0 9 2 0 1 0 0 0 1 0 0 10 61 82 1 2 97 0 0 3 31940 248 0 10 20 0 26 0 27 0 4 0 0 53 189 191 6 6 88 0 0 3 32080 288 3 19 49 6 26 0 15 0 9 0 0 75 415 277 6 15 79 0 0 3 32080 256 0 26 20 6 21 0 12 1 6 0 0 163 110 138 1 3 96 0 1 3 32060 256 3 45 52 28 61 0 27 5 12 0 0 195 191 223 7 11 82 0 0 3 32056 260 0 1 0 0 0 0 0 0 0 0 0 4 52 84 0 1 99 |
Run the vmstat -s command to show the total of various system events that have taken place since the last time the system was booted.
$ vmstat -s 0 swap ins 0 swap outs 0 pages swapped in 0 pages swapped out 392182 total address trans. faults taken 20419 page ins 923 page outs 30072 pages paged in 9194 pages paged out 65167 total reclaims 65157 reclaims from free list 0 micro (hat) faults 392182 minor (as) faults 19383 major faults 85775 copy-on-write faults 66637 zero fill page faults 46309 pages examined by the clock daemon 6 revolutions of the clock hand 15578 pages freed by the clock daemon 4398 forks 352 vforks 4267 execs 12926285 cpu context switches 109029866 device interrupts 499296 traps 22461261 system calls 778068 total name lookups (cache hits 97%) 18739 user cpu 34662 system cpu 52051435 idle cpu 25252 wait cpu |
Run vmstat -S to show swapping statistics.
$ vmstat -S procs memory page disk faults cpu r b w swap free si so pi po fr de sr f0 s0 s6 -- in sy cs us sy id 0 0 0 200968 17936 0 0 0 0 0 0 0 0 0 0 0 109 43 24 0 0 100 |
The swapping statistics fields are described in the following table. For a description of the other fields, see Table 24–1.
Table 24–2 Output From the vmstat -S Command
Field Name |
Description |
---|---|
si |
Average number of LWPs that are swapped in per second |
so |
Number of whole processes that are swapped out |
The vmstat command truncates the output of both fields. Use the sar command to display a more accurate accounting of swap statistics.
Run the vmstat -c command to show cache flushing statistics for a virtual cache.
$ vmstat -c usr ctx rgn seg pag par 0 60714 5 134584 4486560 4718054 |
The output shows the total number of cache flushes since the last boot. The cache types are described in the following table.
Table 24–3 Output From the vmstat -c Command
Cache Name |
Cache Type |
---|---|
usr |
User |
ctx |
Context |
rgn |
Region |
seg |
Segment |
pag |
Page |
par |
Partial-page |
Run the vmstat -i command to show the number of interrupts per device.
The following example shows output from the vmstat -i command.
$ vmstat -i interrupt total rate -------------------------------- clock 52163269 100 esp0 2600077 4 zsc0 25341 0 zsc1 48917 0 cgsixc0 459 0 lec0 400882 0 fdc0 14 0 bppc0 0 0 audiocs0 0 0 -------------------------------- Total 55238959 105 |
Use the iostat command to report statistics about disk input and output, and produces measures of throughput, utilization, queue lengths, transaction rates, and service time. For a detailed description of this command, refer to iostat(1M).
You can display disk utilization information by using the iostat command with a time interval in seconds.
$ iostat 5 tty fd0 sd3 nfs1 nfs31 cpu tin tout kps tps serv kps tps serv kps tps serv kps tps serv us sy wt id 0 1 0 0 410 3 0 29 0 0 9 3 0 47 4 2 0 94 |
The first line of output shows the statistics since the last time the system was booted. Each subsequent line shows the interval statistics. The default is to show statistics for the terminal (tty), disks (fd and sd), and CPU (cpu).
The following table describes the fields in the iostat command output.
Table 24–4 Output From the iostat n Command
Device Type |
Field Name |
Description |
---|---|---|
Terminal |
|
|
|
tin |
Number of characters in the terminal input queue |
|
tout |
Number of characters in the terminal output queue |
Disk |
|
|
|
bps |
Blocks per second |
|
tps |
Transactions per second |
|
serv |
Average service time, in milliseconds |
CPU |
|
|
|
us |
In user mode |
|
sy |
In system mode |
|
wt |
Waiting for I/O |
|
id |
Idle |
The following example shows disk statistics that were gathered every five seconds.
$ iostat 5 tty sd0 sd6 nfs1 nfs49 cpu tin tout kps tps serv kps tps serv kps tps serv kps tps serv us sy wt id 0 0 1 0 49 0 0 0 0 0 0 0 0 15 0 0 0 100 0 47 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 44 6 132 0 0 0 0 0 0 0 0 0 0 0 1 99 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 3 1 23 0 0 0 0 0 0 0 0 0 0 0 1 99 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 |
Run the iostat -xtc command to get extended disk statistics.
$ iostat -xtc extended device statistics tty cpu device r/s w/s kr/s kw/s wait actv svc_t %w %b tin tout us sy wt id fd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 0 100 sd0 0.0 0.0 0.4 0.4 0.0 0.0 49.5 0 0 sd6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 nfs1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 nfs49 0.0 0.0 0.0 0.0 0.0 0.0 15.1 0 0 nfs53 0.0 0.0 0.4 0.0 0.0 0.0 24.5 0 0 nfs54 0.0 0.0 0.0 0.0 0.0 0.0 6.3 0 0 nfs55 0.0 0.0 0.0 0.0 0.0 0.0 4.9 0 0 |
The iostat -xtc command displays a line of output for each disk. The output fields are described in the following table.
Table 24–5 Output From the iostat -xtc Command
Field Name |
Description |
---|---|
r/s |
Reads per second |
w/s |
Writes per second |
kr/s |
Kbytes read per second |
kw/s |
Kbytes written per second |
wait |
Average number of transactions that are waiting for service (queue length) |
actv |
Average number of transactions that are actively being serviced |
svc_t |
Average service time, in milliseconds |
%w |
Percentage of time that the queue is not empty |
%b |
Percentage of time that the disk is busy |
Use the df command to show the amount of free disk space on each mounted disk. The usable disk space that is reported by df reflects only 90 percent of full capacity, as the reporting statistics leave 10 percent above the total available space. This head room normally stays empty for better performance.
The percentage of disk space actually reported by the df command is used space divided by usable space.
If the file system exceeds 90 percent capacity, you could transfer files to a disk that is not as full by using the cp command, or to a tape by using the tar or cpio commands. Or, you could remove the files.
For a detailed description of this command, see df(1M).
Use the df -k command to display disk space information in Kbytes.
$ df -k Filesystem kbytes used avail capacity Mounted on /dev/dsk/c0t3d0s0 192807 40231 133296 24% / |
The following table describes the df -k command output.
Table 24–6 Output From the df -k Command
Field Name |
Description |
---|---|
kbytes |
Total size of usable space in the file system |
used |
Amount of space used |
avail |
Amount of space available for use |
capacity |
Amount of space used, as a percentage of the total capacity |
mounted on |
Mount point |
The following example shows the df -k command output.
$ df -k Filesystem kbytes used avail capacity Mounted on /dev/dsk/c0t0d0s0 384120 131596 214112 39% / /dev/dsk/c0t0d0s6 1388419 1050390 282493 79% /usr /proc 0 0 0 0% /proc mnttab 0 0 0 0% /etc/mnttab fd 0 0 0 0% /dev/fd swap 467152 40 467112 1% /var/run swap 467160 48 467112 1% /tmp /dev/dsk/c0t0d0s4 1784644 1525360 205745 89% /export venus:/usr/dist 20612581 13237316 6963015 66% /usr/dist |
Use the sar command to do the following:
Organize and view data about system activity
Access system activity data on a special request basis
Generate automatic reports to measure and monitor system performance, and special request reports to pinpoint specific performance problems. Collecting System Activity Data Automatically (sar) describes these tools.
For a detailed description of this command, see sar(1).
Display file access operation statistics with the sar -a command.
$ sar -a SunOS touchstone 5.9 Generic sun4u 03/04/2003 00:00:00 iget/s namei/s dirbk/s 01:00:00 0 3 0 02:00:00 0 3 0 03:00:00 0 3 0 04:00:00 0 3 0 05:00:00 0 3 0 06:00:00 0 3 0 07:00:00 0 3 0 08:00:00 0 3 0 08:20:01 0 3 0 08:40:00 0 3 0 09:00:00 0 3 0 09:20:01 0 10 0 09:40:01 0 1 0 10:00:02 0 5 0 Average 0 4 0 |
The following table describes the operating system routines that are reported by the sar -a command.
Table 24–7 Output from the sar -a Command
Field Name |
Description |
---|---|
iget/s |
The number of requests made for inodes that were not in the directory name look-up cache (DNLC). |
namei/s |
The number of file system path searches per second. If namei does not find a directory name in the DNLC, it calls iget to get the inode for either a file or directory. Hence, most igets are the result of DNLC misses. |
dirbk/s |
The number of directory block reads issued per second. |
The larger the reported values, the more time the kernel is spending to access user files. The amount of time reflects how heavily programs and applications are using the file systems. The -a option is helpful for viewing how disk-dependent an application is.
Display buffer activity statistics with the sar -b command.
The buffer is used to cache metadata, which includes inodes, cylinder group blocks, and indirect blocks.
$ sar -b 00:00:00 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s 01:00:00 0 0 100 0 0 55 0 0 |
The following table describes the buffer activities that are displayed by the -b option.
Table 24–8 Output From the sar -b Command
Field Name |
Description |
---|---|
bread/s |
Average number of reads per second that are submitted to the buffer cache from the disk |
lread/s |
Average number of logical reads per second from the buffer cache |
%rcache |
Fraction of logical reads that are found in the buffer cache (100% minus the ratio of bread/s to lread/s) |
bwrit/s |
Average number of physical blocks (512 blocks) that are written from the buffer cache to disk, per second |
lwrit/s |
Average number of logical writes to the buffer cache, per second |
%wcache |
Fraction of logical writes that are found in the buffer cache (100% minus the ratio of bwrit/s to lwrit/s) |
pread/s |
Average number of physical reads, per second that use character device interfaces |
pwrit/s |
Average number of physical write requests, per second that use character device interfaces |
The most important entries are the cache hit ratios %rcache and %wcache, which measure the effectiveness of system buffering. If %rcache falls below 90 percent, or if %wcache falls below 65 percent, it might be possible to improve performance by increasing the buffer space.
The following example of sar -b output shows that the %rcache and %wcache buffers are not causing any slowdowns. All the data is within acceptable limits.
$ sar -b SunOS touchstone 5.9 Generic sun4u 03/04/2003 00:00:00 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s 01:00:00 0 0 100 0 0 50 0 0 02:00:00 0 0 100 0 0 45 0 0 03:00:00 0 0 100 0 0 72 0 0 04:00:00 0 0 100 0 0 43 0 0 05:00:00 0 0 100 0 0 42 0 0 06:00:00 0 0 100 0 0 48 0 0 07:00:00 0 0 100 0 0 42 0 0 08:00:00 0 0 100 0 0 48 0 0 08:20:01 0 0 100 0 0 42 0 0 08:40:00 0 0 100 0 0 48 0 0 09:00:00 0 0 100 0 0 43 0 0 09:20:01 0 2 99 0 2 83 0 0 09:40:01 0 0 98 0 0 61 0 0 10:00:02 0 1 99 0 0 62 0 0 Average 0 0 99 0 0 74 0 0 |
Display system call statistics by using the sar -c command.
$ sar -c 00:00:00 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s 01:00:00 38 2 2 0.00 0.00 149 120 |
The following table describes the system call categories that are reported by the -c option. Typically, reads and writes account for about half of the total system calls, although the percentage varies greatly with the activities that are being performed by the system.
Table 24–9 Output From the sar -c Command
Field Name |
Description |
---|---|
scall/s |
All types of system calls per second, which is generally about 30 per second on a system with 4 to 6 users. |
sread/s |
read system calls per second. |
swrit/s |
write system calls per second. |
fork/s |
fork system calls per second, which is about 0.5 per second on a system with 4 to 6 users. This number will increase if shell scripts are running. |
exec/s |
exec system calls per second. If exec/s divided by fork/s is greater than three, look for inefficient PATH variables. |
rchar/s |
Characters (bytes) transferred by read system calls per second. |
wchar/s |
Characters (bytes) transferred by write system calls per second. |
The following example shows output from the sar -c command.
$ sar -c SunOS touchstone 5.9 Generic sun4u 03/04/2003 00:00:00 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s 01:00:00 118 14 9 0.00 0.00 3360 2506 02:00:00 118 14 9 0.00 0.00 3360 2505 03:00:00 118 14 9 0.00 0.00 3361 2506 04:00:00 118 14 9 0.01 0.00 3363 2506 05:00:00 118 14 9 0.00 0.00 3360 2505 06:00:00 118 14 9 0.00 0.00 3360 2505 07:00:00 118 14 9 0.00 0.00 3360 2505 08:00:00 118 14 9 0.00 0.00 3360 2505 08:20:01 120 14 9 0.00 0.00 3394 2522 08:40:00 119 14 9 0.01 0.01 3357 2502 09:00:00 119 14 9 0.01 0.00 3356 2500 09:20:01 437 76 48 0.04 0.04 19606 6531 09:40:01 585 117 73 0.00 0.00 12962 10119 10:00:02 564 97 60 0.03 0.03 16354 6196 Average 159 22 14 0.01 0.01 4657 3017 |
Display disk activity statistics with the sar -d command.
$ sar -d 00:00:00 device %busy avque r+w/s blks/s avwait avserv 01:00:00 fd0 0 0.0 0 0 0.0 0.0 |
The following table describes the disk device activities that are reported by the -d option.
Table 24–10 Output From the sar -d Command
Field Name |
Description |
---|---|
device |
Name of the disk device that is being monitored |
%busy |
Percentage of time the device spent servicing a transfer request |
avque |
The sum of the average wait time plus the average service time |
r+w/s |
Number of read and write transfers to the device, per second |
blks/s |
Number of 512-byte blocks that are transferred to the device, per second |
avwait |
Average time, in milliseconds, that transfer requests wait idly in the queue. This time is measured only when the queue is occupied. |
avserv |
Average time, in milliseconds, for a transfer request to be completed by the device. For disks, this value includes seek, rotational latency, and data transfer times. |
Note that queue lengths and wait times are measured when something is in the queue. If %busy is small, large queues and service times probably represent the periodic efforts by the system to ensure that altered blocks are promptly written to the disk.
This abbreviated example illustrates the sar -d output.
$ sar -d SunOS touchstone 5.9 Generic sun4u 03/04/2003 00:00:00 device %busy avque r+w/s blks/s avwait avserv 01:00:00 dad0 0 0.0 0 0 18.1 14.9 dad0,a 0 0.0 0 0 0.0 0.0 dad0,b 0 0.0 0 0 0.0 15.1 dad0,c 0 0.0 0 0 0.0 0.0 dad0,d 0 0.0 0 0 20.1 14.9 dad0,e 0 0.0 0 0 0.0 0.0 dad0,g 0 0.0 0 0 0.0 0.0 dad0,h 0 0.0 0 0 0.0 0.0 fd0 0 0.0 0 0 0.0 0.0 nfs1 0 0.0 0 0 0.0 0.0 nfs2 0 0.0 0 0 2.9 4.4 nfs3 0 0.0 0 0 0.0 0.0 sd0 0 0.0 0 0 0.0 0.0 |
Use the sar -g command to display page-out and memory freeing activities in averages.
$ sar -g 00:00:00 pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf 01:00:00 0.00 0.00 0.00 0.00 0.00 |
The output displayed by the sar -g command is a good indicator of whether more memory might be needed. Use the ps -elf command to show the number of cycles that are used by the page daemon. A high number of cycles, combined with high values for pgfree/s and pgscan/s, indicates a memory shortage.
The sar -g command also shows whether inodes are being recycled too quickly and causing a loss of reusable pages.
The following table describes the output from the -g option.
Table 24–11 Output From the sar -g Command
Field Name |
Description |
---|---|
pgout/s |
The number of page-out requests per second. |
ppgout/s |
The actual number of pages that are paged-out, per second. A single page-out request might involve paging-out multiple pages. |
pgfree/s |
The number of pages, per second, that are placed on the free list. |
pgscan/s |
The number of pages, per second, that are scanned by the page daemon. If this value is high, the page daemon is spending a lot of time checking for free memory. This situation implies that more memory might be needed. |
%ufs_ipf |
The percentage of ufs inodes taken off the free list by iget that had reusable pages associated with them. These pages are flushed and cannot be reclaimed by processes. Thus, this field represents the percentage of igets with page flushes. A high value indicates that the free list of inodes is page-bound, and the number of ufs inodes might need to be increased. |
The following example shows output from the sar -g command.
$ sar -g SunOS touchstone 5.9 Generic sun4u 03/04/2003 00:00:00 pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf 01:00:00 0.00 0.00 0.00 0.00 0.00 02:00:00 0.01 0.01 0.01 0.00 0.00 03:00:00 0.00 0.00 0.00 0.00 0.00 04:00:00 0.00 0.00 0.00 0.00 0.00 05:00:00 0.00 0.00 0.00 0.00 0.00 06:00:00 0.00 0.00 0.00 0.00 0.00 07:00:00 0.00 0.00 0.00 0.00 0.00 08:00:00 0.00 0.00 0.00 0.00 0.00 08:20:01 0.00 0.00 0.00 0.00 0.00 08:40:00 0.00 0.00 0.00 0.00 0.00 09:00:00 0.00 0.00 0.00 0.00 0.00 09:20:01 0.05 0.52 1.62 10.16 0.00 09:40:01 0.03 0.44 1.47 4.77 0.00 10:00:02 0.13 2.00 4.38 12.28 0.00 10:20:03 0.37 4.68 12.26 33.80 0.00 Average 0.02 0.25 0.64 1.97 0.00 |
Use the sar -k command to report on the following activities of the Kernel Memory Allocator (KMA).
The KMA allows a kernel subsystem to allocate and free memory as needed. Rather than statically allocating the maximum amount of memory it is expected to require under peak load, the KMA divides requests for memory into three categories: small (less than 256 bytes), large (512 bytes to 4 Kbytes), and oversized (greater than 4 Kbytes). The KMA keeps two pools of memory to satisfy small and large requests. The oversized requests are satisfied by allocating memory from the system page allocator.
If you are investigating a system that is being used to write drivers or STREAMS that use KMA resources, then the sar -k command will likely prove useful. Otherwise, you will probably not need the information it provides. Any driver or module that uses KMA resources, but does not specifically return the resources before it exits, can create a memory leak. A memory leak causes the amount of memory that is allocated by KMA to increase over time. Thus, if the alloc fields of the sar -k command increase steadily over time, there might be a memory leak. Another indication of a memory leak is failed requests. If this problem occurs, a memory leak has probably caused KMA to be unable to reserve and allocate memory.
If it appears that a memory leak has occurred, you should check any drivers or STREAMS that might have requested memory from KMA and not returned it.
$ sar -k 00:00:00 sml_mem alloc fail lg_mem alloc fail ovsz_alloc fail 01:00:00 2523136 1866512 0 18939904 14762364 0 360448 0 02:00:02 2523136 1861724 0 18939904 14778748 0 360448 0 |
The following table describes the output from the -k option.
Table 24–12 Output From the sar -k Command
Field Name |
Description |
---|---|
sml_mem |
The amount of memory, in bytes, that the KMA has available in the small memory request pool, where a small request is less than 256 bytes. |
alloc |
The amount of memory, in bytes, that the KMA has allocated from its small memory request pool to small memory requests. |
fail |
The number of requests for small amounts of memory that failed. |
lg_mem |
The amount of memory, in bytes, that the KMA has available in the large memory request pool, where a large request is from 512 bytes to 4 Kbytes. |
alloc |
The amount of memory, in bytes, that the KMA has allocated from its large memory request pool to large memory requests. |
fail |
The number of failed requests for large amounts of memory. |
ovsz_alloc |
The amount of memory that is allocated for oversized requests, which are requests that are greater than 4 Kbytes. These requests are satisfied by the page allocator. Thus, there is no pool. |
fail |
The number of failed requests for oversized amounts of memory. |
The following is an abbreviated example of sar -k output.
$ sar -k SunOS touchstone 5.9 Generic sun4u 03/04/2003 00:00:05 sml_mem alloc fail lg_mem alloc fail ovsz_alloc fail 01:00:03 1761280 1209760 0 4833280 3955104 0 2293760 0 02:00:05 1761280 1209020 0 4833280 3953360 0 2293760 0 03:00:03 1761280 1208712 0 4833280 3972096 0 2293760 0 04:00:03 1761280 1209796 0 4833280 3976432 0 2293760 0 05:00:03 1761280 1207708 0 4833280 3976864 0 2293760 0 06:00:04 1761280 1208112 0 4833280 3977200 0 2293760 0 07:00:04 1761280 1209024 0 4833280 3977200 0 2293760 0 08:00:04 1761280 1208932 0 4833280 4035120 0 2293760 0 08:20:04 1761280 1210480 0 4833280 4035120 0 2293760 0 08:40:04 1761280 1210592 0 4833280 4035968 0 2293760 0 09:00:03 1761280 1212440 0 4833280 4039024 0 2293760 0 09:20:04 1761280 1214388 0 4833280 4043792 0 2293760 0 09:40:04 1761280 1210760 0 4833280 4041744 0 2293760 0 10:00:05 1761280 1208192 0 4833280 4040496 0 2293760 0 10:20:03 1761280 1213404 0 4833280 4046576 0 2293760 0 10:40:02 1761280 1212184 0 4833280 4046576 0 2293760 0 Average 1761280 1210219 0 4833280 4009542 0 2293760 0 |
Use the sar -m command to report interprocess communication activities.
$ sar -m 00:00:00 msg/s sema/s 01:00:00 0.00 0.00 |
These figures will usually be zero (0.00), unless you are running applications that use messages or semaphores.
The following table describes the output from the -m option.
Table 24–13 Output From the sar -m Command
Field Name |
Description |
---|---|
msg/s |
The number of message operations (sends and receives) per second |
sema/s |
The number of semaphore operations per second |
The following abbreviated example shows output from the sar -m command.
$ sar -m SunOS touchstone 5.9 Generic sun4u 03/04/2003 00:00:00 msg/s sema/s 01:00:00 0.00 0.00 02:00:02 0.00 0.00 03:00:00 0.00 0.00 04:00:00 0.00 0.00 05:00:01 0.00 0.00 06:00:00 0.00 0.00 Average 0.00 0.00 |
Use the sar -p command to report page-in activity, which includes protection and translation faults.
$ sar -p 00:00:00 atch/s pgin/s ppgin/s pflt/s vflt/s slock/s 01:00:00 0.07 0.00 0.00 0.21 0.39 0.00 |
The following table describes the reported statistics from the -p option.
Table 24–14 Output From the sar -p Command
Field Name |
Description |
---|---|
atch/s |
The number of page faults, per second, that are satisfied by reclaiming a page currently in memory (attaches per second). Instances include reclaiming an invalid page from the free list and sharing a page of text that is currently being used by another process. For example, two or more processes that are accessing the same program text. |
pgin/s |
The number of times, per second, that file systems receive page-in requests. |
ppgin/s |
The number of pages paged in, per second. A single page-in request, such as a soft-lock request (see slock/s), or a large block size, might involve paging-in multiple pages. |
pflt/s |
The number of page faults from protection errors. Instances of protection faults are illegal access to a page and “copy-on-writes.” Generally, this number consists primarily of “copy-on-writes.” |
vflt/s |
The number of address translation page faults, per second. These faults are known as validity faults, and occur when a valid process table entry does not exist for a given virtual address. |
slock/s |
The number of faults, per second, caused by software lock requests that require physical I/O. An example of the occurrence of a soft-lock request is the transfer of data from a disk to memory. The system locks the page that is to receive the data, so that the page cannot be claimed and used by another process. |
The following example shows output from the sar -p command.
$ sar -p SunOS touchstone 5.9 Generic sun4u 03/04/2003 00:00:00 atch/s pgin/s ppgin/s pflt/s vflt/s slock/s 01:00:00 1.00 0.00 0.00 0.26 1.13 0.00 02:00:00 1.00 0.00 0.00 0.26 1.13 0.00 03:00:00 1.01 0.00 0.00 0.29 1.17 0.00 04:00:00 1.03 0.00 0.00 0.34 1.27 0.00 05:00:00 1.00 0.00 0.00 0.26 1.13 0.00 06:00:00 1.00 0.00 0.00 0.26 1.13 0.00 07:00:00 1.00 0.00 0.00 0.26 1.13 0.00 08:00:00 1.00 0.00 0.00 0.26 1.13 0.00 08:20:01 1.03 0.00 0.00 0.32 1.26 0.00 08:40:00 1.03 0.00 0.00 0.32 1.26 0.00 09:00:00 1.03 0.00 0.00 0.42 1.43 0.00 09:20:01 2.17 1.37 1.95 2.59 50.06 0.00 09:40:01 0.09 0.52 0.63 0.27 105.42 0.00 10:00:02 0.65 2.59 3.11 1.19 42.19 0.00 10:20:03 1.69 11.45 13.77 4.36 39.58 0.00 10:40:01 0.08 0.31 0.32 1.56 12.23 0.00 Average 1.00 0.51 0.62 0.55 8.79 0.00 |
Use the sar -q command to report the average queue length while the queue is occupied, and the percentage of time that the queue is occupied.
$ sar -q 00:00:00 runq-sz %runocc swpq-sz %swpocc |
The following table describes the output from the -q option.
Table 24–15 Output From the sar -q Command
Field Name |
Description |
---|---|
runq-sz |
The number of kernel threads in memory that are waiting for a CPU to run. Typically, this value should be less than 2. Consistently higher values mean that the system might be CPU-bound. |
%runocc |
The percentage of time that the dispatch queues are occupied. |
swpq-sz |
No longer reported by sar. |
%swpocc |
No longer reported by sar. |
The following example shows output from the sar -q command. If %runocc is high (greater than 90 percent) and runq-sz is greater than 2, the CPU is heavily loaded and response is degraded. In this case, additional CPU capacity might be required to obtain acceptable system response.
$ sar -q SunOS touchstone 5.9 Generic sun4u 03/04/2003 00:00:00 runq-sz %runocc swpq-sz %swpocc 01:00:00 0.0 0 0.0 0 02:00:00 0.0 0 0.0 0 03:00:00 0.0 0 0.0 0 04:00:00 1.0 0 0.0 0 05:00:00 0.0 0 0.0 0 06:00:00 0.0 0 0.0 0 07:00:00 1.0 0 0.0 0 08:00:00 0.0 0 0.0 0 08:20:01 0.0 0 0.0 0 08:40:00 0.0 0 0.0 0 09:00:00 1.0 0 0.0 0 09:20:01 1.6 3 0.0 0 09:40:01 1.5 8 0.0 0 10:00:02 1.6 7 0.0 0 10:20:03 1.5 2 0.0 0 10:40:01 1.9 1 0.0 0 11:00:01 1.4 0 0.0 0 11:20:01 1.6 0 0.0 0 Average 1.6 1 0.0 0 |
Use the sar -r command to report the number of memory pages and swap-file disk blocks that are currently unused.
$ sar -r 00:00:00 freemem freeswap 01:00:00 2135 401922 |
The following table describes the output from the -r option.
Table 24–16 Output From the sar -r Command
Field Name |
Description |
---|---|
freemem |
The average number of memory pages that are available to user processes over the intervals sampled by the command. Page size is machine-dependent. |
freeswap |
The number of 512-byte disk blocks that are available for page swapping. |
The following example shows output from the sar -r command.
$ sar -r SunOS touchstone 5.9 Generic sun4u 03/04/2003 00:00:00 freemem freeswap 01:00:00 3454 834587 02:00:00 3449 834549 03:00:00 3446 834552 04:00:00 3440 834722 05:00:00 3433 834535 06:00:00 3432 834557 07:00:00 3426 834526 08:00:00 3423 834502 08:20:01 3425 835158 08:40:00 3421 834453 09:00:00 3422 834473 09:20:01 2455 817457 09:40:01 651 768238 10:00:02 862 748302 10:20:03 373 713652 10:40:01 506 717459 11:00:01 668 716669 11:20:01 537 715802 Average 2907 815627 |
Use the sar -u command to display CPU utilization statistics.
$ sar -u 00:00:00 %usr %sys %wio %idle 01:00:00 0 0 0 100 |
The sar command without any options is equivalent to sar -u. At any given moment, the processor is either busy or idle. When busy, the processor is in either user mode or system mode. When idle, the processor is either waiting for I/O completion or “sitting still” with no work to do.
The following table describes output from the -u option.
Table 24–17 Output From the sar -u Command
Field Name |
Description |
---|---|
%usr |
Lists the percentage of time that the processor is in user mode |
%sys |
Lists the percentage of time that the processor is in system mode |
%wio |
Lists the percentage of time that the processor is idle and waiting for I/O completion |
%idle |
Lists the percentage of time that the processor is idle and not waiting for I/O |
A high %wio generally means that a disk slowdown has occurred.
The following example shows output from the sar -u command.
$ sar -u SunOS touchstone 5.9 Generic sun4u 03/04/2003 00:00:00 %usr %sys %wio %idle 01:00:00 0 0 0 100 02:00:00 0 0 0 100 03:00:00 0 0 0 100 04:00:00 0 0 0 100 05:00:00 0 0 0 100 06:00:00 0 0 0 100 07:00:00 0 0 0 100 08:00:00 0 0 0 100 08:20:01 0 0 0 100 08:40:00 0 0 0 100 09:00:00 0 0 0 100 09:20:01 6 1 1 91 09:40:01 15 2 0 82 10:00:02 13 1 2 83 10:20:03 5 2 8 84 10:40:01 3 1 0 96 11:00:01 1 0 1 97 11:20:01 2 0 0 97 Average 2 0 0 98 |
Use the sar -v command to report the status of the process table, inode table, file table, and shared memory record table.
$ sar -v 00:00:00 proc-sz ov inod-sz ov file-sz ov lock-sz 01:00:00 43/922 0 2984/4236 0 322/322 0 0/0 |
Output from the -v option is described in the following table.
Table 24–18 Output From the sar -v Command
Field Name |
Description |
---|---|
proc-sz |
The number of process entries (proc structures) that are currently being used, or allocated in the kernel. |
inod-sz |
The total number of inodes in memory versus the maximum number of inodes that are allocated in the kernel. This number is not a strict high water mark. The number can overflow. |
file-sz |
The size of the open system file table. The sz is given as 0, since space is allocated dynamically for the file table. |
ov |
The number of shared memory record table entries that are currently being used or allocated in the kernel. The sz is given as 0 because space is allocated dynamically for the shared memory record table. |
lock-sz |
The number of shared memory record table entries that are currently being used or allocated in the kernel. The sz is given as 0 because space is allocated dynamically for the shared memory record table. |
The following abbreviated example shows output from the sar -v command. This example shows that all tables are large enough to have no overflows. These tables are all dynamically allocated based on the amount of physical memory.
$ sar -v SunOS touchstone 5.9 Generic sun4u 03/04/2003 00:00:00 proc-sz ov inod-sz ov file-sz ov lock-sz 01:00:00 64/1898 0 4280/8467 0 0/0 0 0/0 02:00:00 64/1898 0 4280/8467 0 0/0 0 0/0 03:00:00 64/1898 0 4280/8467 0 0/0 0 0/0 04:00:00 64/1898 0 4280/8467 0 0/0 0 0/0 05:00:00 64/1898 0 4280/8467 0 0/0 0 0/0 06:00:00 64/1898 0 4280/8467 0 0/0 0 0/0 07:00:00 64/1898 0 4280/8467 0 0/0 0 0/0 08:00:00 64/1898 0 4280/8467 0 0/0 0 0/0 08:20:01 64/1898 0 4280/8467 0 0/0 0 0/0 08:40:00 64/1898 0 4280/8467 0 0/0 0 0/0 09:00:00 64/1898 0 4280/8467 0 0/0 0 0/0 09:20:01 70/1898 0 4281/8467 0 0/0 0 0/0 09:40:01 70/1898 0 4281/8467 0 0/0 0 0/0 10:00:02 71/1898 0 4285/8467 0 0/0 0 0/0 |
Use the sar -w command to report swapping and switching activity.
$ sar -w 00:00:00 swpin/s bswin/s swpot/s bswot/s pswch/s 01:00:00 0.00 0.0 0.00 0.0 22 |
The following table describes target values and observations.
Table 24–19 Output From the sar -w Command
Field Name |
Description |
---|---|
swpin/s |
The number of LWP transfers into memory per second. |
bswin/s |
The average number of processes that are swapped out of memory per second. If the number is greater than 1, you might need to increase memory. |
swpot/s |
The average number of processes that are swapped out of memory per second. If the number is greater than 1, you might need to increase memory. |
bswot/s |
The number of blocks that are transferred for swap-outs per second. |
pswch/s |
The number of kernel thread switches, per second. |
All process swap-ins include process initialization.
The following example shows output from the sar -w command.
$ sar -w SunOS touchstone 5.9 Generic sun4u 03/04/2003 00:00:00 swpin/s bswin/s swpot/s bswot/s pswch/s 01:00:00 0.00 0.0 0.00 0.0 63 02:00:00 0.00 0.0 0.00 0.0 60 03:00:00 0.00 0.0 0.00 0.0 62 04:00:00 0.00 0.0 0.00 0.0 63 05:00:00 0.00 0.0 0.00 0.0 63 06:00:00 0.00 0.0 0.00 0.0 61 07:00:00 0.00 0.0 0.00 0.0 60 08:00:00 0.00 0.0 0.00 0.0 61 08:20:01 0.00 0.0 0.00 0.0 63 08:40:00 0.00 0.0 0.00 0.0 63 09:00:00 0.00 0.0 0.00 0.0 63 09:20:01 0.00 0.0 0.00 0.0 132 09:40:01 0.00 0.0 0.00 0.0 187 10:00:02 0.00 0.0 0.00 0.0 185 10:20:03 0.00 0.0 0.00 0.0 258 10:40:01 0.00 0.0 0.00 0.0 161 11:00:01 0.00 0.0 0.00 0.0 128 11:20:01 0.00 0.0 0.00 0.0 117 11:40:00 0.00 0.0 0.00 0.0 171 12:00:01 0.00 0.0 0.00 0.0 94 12:20:01 0.00 0.0 0.00 0.0 95 12:40:01 0.00 0.0 0.00 0.0 95 13:00:01 0.00 0.0 0.00 0.0 95 13:20:01 0.00 0.0 0.00 0.0 96 13:40:01 0.00 0.0 0.00 0.0 98 14:00:01 0.00 0.0 0.00 0.0 96 14:20:00 0.00 0.0 0.00 0.0 91 14:40:01 0.00 0.0 0.00 0.0 92 15:00:01 0.00 0.0 0.00 0.0 103 Average 0.00 0.0 0.00 0.0 88 |
Use the sar -y command to monitor terminal device activities.
$ sar -y 00:00:00 rawch/s canch/s outch/s rcvin/s xmtin/s mdmin/s 01:00:00 0 0 0 0 0 0 |
If you have a lot of terminal I/O, you can use this report to determine if there are any bad lines. The activities recorded are defined in the following table.
Table 24–20 Output From the sar -y Command
Field Name |
Description |
---|---|
rawch/s |
Input characters (raw queue), per second |
canch/s |
Input characters that are processed by canon (canonical queue), per second |
outch/s |
Output characters (output queue) per second |
rcvin/s |
Receiver hardware interrupts per second |
xmtin/s |
Transmitter hardware interrupts per second |
mdmin/s |
Modem interrupts per second |
The number of modem interrupts per second (mdmin/s) should be close to zero. The receive and transmit interrupts per second (xmtin/s and rcvin/s) should be less than or equal to the number of incoming or outgoing characters, respectively. If not, check for bad lines.
The following example shows output from the sar -y command.
$ sar -y SunOS touchstone 5.9 Generic sun4u 03/04/2003 00:00:00 rawch/s canch/s outch/s rcvin/s xmtin/s mdmin/s 01:00:00 0 0 0 0 0 0 02:00:00 0 0 0 0 0 0 03:00:00 0 0 0 0 0 0 04:00:00 0 0 0 0 0 0 05:00:00 0 0 0 0 0 0 06:00:00 0 0 0 0 0 0 07:00:00 0 0 0 0 0 0 08:00:00 0 0 0 0 0 0 08:20:01 0 0 0 0 0 0 08:40:00 0 0 0 0 0 0 09:00:00 0 0 0 0 0 0 09:20:01 0 0 6 0 0 0 09:40:01 0 0 0 0 0 0 10:00:02 0 0 1 0 0 0 10:20:03 0 0 6 0 0 0 10:40:01 0 0 15 0 0 0 11:00:01 0 0 20 0 0 0 11:20:01 0 0 4 0 0 0 11:40:00 0 0 4 0 0 0 12:00:01 0 0 0 0 0 0 Average 0 0 2 0 0 0 |
Use the sar -A command to display statistics from all options to provide a view of overall system performance.
This command provides a more global perspective. If data from more than a single time segment is shown, the report includes averages.
Three commands are involved in the automatic collection of system activity data: sadc, sa1, and sa2.
The sadc data collection utility periodically collects data on system activity and saves the data in a file in binary format, one file for each 24-hour period. You can set up the sadc command to run periodically (usually once each hour), and whenever the system boots to multiuser mode. The data files are placed in the /var/adm/sa directory. Each file is named sadd, where dd is the current date. The format of the command is as follows:
/usr/lib/sa/sadc [t n] [ofile] |
The command samples n times with an interval of t seconds, which should be greater than 5 seconds between samples. This command then writes to the binary ofile file, or to standard output.
The sadc command should be run at system boot time in order to record the statistics from when the counters are reset to zero. To make sure that sadc is run at boot time, the /etc/init.d/perf file contains a command line that writes a record to the daily data file.
The command entry has the following format:
/usr/bin/su sys -c "/usr/lib/sa/sadc /var/adm/sa/sa`date +%d`" |
To generate periodic records, you need to run the sadc command regularly. The simplest way to do so is uncomment the following lines in the /var/spool/cron/crontab/sys file:
# 0 * * * 0-6 /usr/lib/sa/sa1 # 20,40 8-17 * * 1-5 /usr/lib/sa/sa1 # 5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A |
The sys crontab entries do the following:
The first two crontab entries cause a record to be written to the /var/adm/sa/sadd file every 20 minutes from 8 a.m. to 5 p.m., Monday through Friday, and every hour on the hour otherwise.
The third entry writes a record to the /var/adm/sa/sardd file hourly, Monday through Friday, and includes all sar options.
You can change these defaults to meet your needs.
Another shell script, sa2, produces reports rather than binary data files. The sa2 command invokes the sar command and writes the ASCII output to a report file.
The sar command can be used either to gather system activity data itself or to report what has been collected in the daily activity files that are created by the sadc command.
The sar command has the following formats:
sar [-aAbcdgkmpqruvwy] [-o file] t [n] |
sar [-aAbcdgkmpqruvwy] [-s time] [-e time] [-i sec] [-f file] |
The following sar command samples cumulative activity counters in the operating system every t seconds, n times. The t should be 5 seconds or greater. Otherwise, the command itself might affect the sample. You must specify a time interval between which to take the samples. Otherwise, the command operates according to the second format. The default value of n is 1. The following example takes two samples separated by 10 seconds. If the -o option is specified, samples are saved in binary format.
$ sar -u 10 2 |
Other important information about the sar command includes the following:
With no sampling interval or number of samples specified, the sar command extracts data from a previously recorded file, either the file specified by the -f option or, by default, the standard daily activity file, /var/adm/sa/sadd, for the most recent day.
The -s and -e options define the starting and ending times for the report. Starting and ending times are of the form hh[:mm[:ss]], where h, m, and s represent hours, minutes, and seconds.
The -i option specifies, in seconds, the intervals between record selection. If the -i option is not included, all intervals that are found in the daily activity file are reported.
The following table lists the sar options and their actions.
Table 24–21 Options for the sar Command
Option |
Actions |
---|---|
-a |
Checks file access operations |
-b |
Checks buffer activity |
-c |
Checks system calls |
-d |
Checks activity for each block device |
-g |
Checks page-out and memory freeing |
-k |
Checks kernel memory allocation |
-m |
Checks interprocess communication |
-p |
Checks swap and dispatch activity |
-q |
Checks queue activity |
-r |
Checks unused memory |
-u |
Checks CPU utilization |
-nv |
Checks system table status |
-w |
Checks swapping and switching volume |
-y |
Checks terminal activity |
-A |
Reports overall system performance, which is the same as entering all options. |
If no option is used, it is equivalent to calling the command with the -u option.
Edit the /etc/init.d/perf file and uncomment the following lines:
# if [ -z "$_INIT_RUN_LEVEL" ]; then # set -- `/usr/bin/who -r` # _INIT_RUN_LEVEL="$7" # _INIT_RUN_NPREV="$8" # _INIT_PREV_LEVEL="$9" # fi # # if [ $_INIT_RUN_LEVEL -ge 2 -a $_INIT_RUN_LEVEL -le 4 -a \ # $_INIT_RUN_NPREV -eq 0 -a \( $_INIT_PREV_LEVEL = 1 -o \ # $_INIT_PREV_LEVEL = S \) ]; then # # /usr/bin/su sys -c "/usr/lib/sa/sadc /var/adm/sa/sa`date +%d`" # fi |
This version of the sadc command writes a special record that marks the time when the counters are reset to zero (boot time).
Edit the /var/spool/cron/crontabs/sys crontab file and uncomment the following lines:
# 0 * * * 0-6 /usr/lib/sa/sa1 # 20,40 8-17 * * 1-5 /usr/lib/sa/sa1 # 5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A |