System Administration Guide, Volume 2

Chapter 36 Monitoring Performance (Tasks)

This chapter describes procedures for monitoring system performance by using the vmstat, iostat, df, and sar commands. This is a list of the step-by-step instructions in this chapter.

Displaying Virtual Memory Statistics (vmstat)

You can use the vmstat command to report virtual memory statistics and such information about system events as CPU load, paging, number of context switches, device interrupts, and system calls. The vmstat command can also display statistics on swapping, cache flushing, and interrupts.

Refer to vmstat(1M) for a more detailed description of this command.

How to Display Virtual Memory Statistics (vmstat)

Collect virtual memory statistics using the vmstat command with a time interval.


$ vmstat n

n

Interval in seconds between reports. 

The table below describes the fields in the vmstat output.

Table 36-1 Output From the vmstat Command

Category 

Field Name 

Description 

procs

 

Reports the following states: 

 

r

The number of kernel threads in the dispatch queue 

 

b

Blocked kernel threads waiting for resources 

 

w

Swapped out LWPs waiting for processing resources to finish  

memory

 

Reports on usage of real and virtual memory: 

 

swap

Available swap space 

 

free

Size of the free list 

page

 

Reports on page faults and paging activity, in units per second: 

 

re

Pages reclaimed 

 

mf

Minor and major faults 

 

pi

Kbytes paged in 

 

po

Kbytes paged out 

 

fr

Kbytes freed 

 

de

Anticipated memory needed by recently swapped-in processes 

 

sr

Pages scanned by page daemon (not currently in use). If sr does not equal zero, the page daemon has been running.

disk

 

Reports the number of disk operations per second, showing data on up to four disks 

faults

 

Reports the trap/interrupt rates (per second): 

 

in

Interrupts per second 

 

sy

System calls per second 

 

cs

CPU context switch rate 

cpu

 

Reports on the use of CPU time: 

 

us

User time 

 

sy

System time 

 

id

Idle time  

Example--Displaying Virtual Memory Statistics

The following example shows the vmstat display of statistics gathered at five-second intervals.


$ vmstat 5
 procs    memory            page             disk      faults     cpu
r b w  swap free re  mf  pi  po  fr de sr f0 s3 -- --  in  sy  cs us sy  id
0 0 8 28312  668  0   9   2   0   1  0  0  0  1  0  0  10  61  82  1  2  97
0 0 3 31940  248  0  10  20   0  26  0 27  0  4  0  0  53 189 191  6  6  88
0 0 3 32080  288  3  19  49   6  26  0 15  0  9  0  0  75 415 277  6 15  79
0 0 3 32080  256  0  26  20   6  21  0 12  1  6  0  0 163 110 138  1  3  96
0 1 3 32060  256  3  45  52  28  61  0 27  5 12  0  0 195 191 223  7 11  82
0 0 3 32056  260  0   1   0   0   0  0  0  0  0  0  0   4  52  84  0  1  99

How to Display System Event Information (vmstat -s)

Run vmstat -s to show the total of various system events that have taken place since the system was last booted.


$ vmstat -s
        0 swap ins
        0 swap outs
        0 pages swapped in
        0 pages swapped out
   392182 total address trans. faults taken
    20419 page ins
      923 page outs
    30072 pages paged in
     9194 pages paged out
    65167 total reclaims
    65157 reclaims from free list
        0 micro (hat) faults
   392182 minor (as) faults
    19383 major faults
    85775 copy-on-write faults
    66637 zero fill page faults
    46309 pages examined by the clock daemon
        6 revolutions of the clock hand
    15578 pages freed by the clock daemon
     4398 forks
      352 vforks
     4267 execs
 12926285 cpu context switches
109029866 device interrupts
   499296 traps
 22461261 system calls
   778068 total name lookups (cache hits 97%)
    18739 user   cpu
    34662 system cpu
 52051435 idle   cpu
    25252 wait   cpu

How to Display Swapping Statistics (vmstat -S)

Run vmstat -S to show swapping statistics.


$ vmstat -S
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  si  so pi po fr de sr f0 s0 s6 --   in   sy   cs us sy id
 0 0 0 200968 17936   0   0  0  0  0  0  0  0  0  0  0  109   43   24  0  0 100

The fields are described in the table below.

Table 36-2 Output From the vmstat -S Command

Field Name 

Description 

si

Average number of LWPs swapped in per second 

so

Number of whole processes swapped out 


Note -

The vmstat command truncates the output of both of these fields. Use the sar command to display a more accurate accounting of swap statistics.


How to Display Cache Flushing Statistics (vmstat -c)

Run vmstat -c to show cache flushing statistics for a virtual cache.


$ vmstat -c
usr     ctx     rgn     seg     pag     par
  0   60714       5  134584 4486560 4718054

It shows the total number of cache flushes since the last boot. The cache types are described in the table below.

Table 36-3 Output From the vmstat -c Command

Cache Name 

Cache Type 

usr

User 

ctx

Context 

rgn

Region 

seg

Segment 

pag

Page 

par

Partial-page  

How to Display Interrupts Per Device (vmstat -i)

Run vmstat -i to show interrupts per device.


$ vmstat -i

Example--Displaying Interrupts Per Device

The following example shows output from the vmstat -i command.


$ vmstat -i
interrupt         total     rate
--------------------------------
clock          52163269      100
esp0            2600077        4
zsc0              25341        0
zsc1              48917        0
cgsixc0             459        0
lec0             400882        0
fdc0                 14        0
bppc0                 0        0
audiocs0              0        0
--------------------------------
Total          55238959      105

Displaying Disk Utilization Information (iostat n)

Use the iostat command to report statistics about disk input and output, and produces measures of throughput, utilization, queue lengths, transaction rates, and service time. For a detailed description of this command, refer to iostat(1M).

How to Display Disk Utilization Information (iostat)

You can display disk activity information by using the iostat command with a time interval.


$ iostat 5
     tty          fd0           sd3          nfs1         nfs31          cpu
tin tout kps tps serv  kps tps serv  kps tps serv  kps tps serv  us sy wt id
  0    1   0   0  410    3   0   29    0   0    9    3   0   47   4  2  0 94

The first line of output shows the statistics since the last boot. Each subsequent line shows the interval statistics. The default is to show statistics for the terminal (tty), disks (fd and sd), and CPU (cpu).

The table below describes the fields in the iostat command output.

Table 36-4 Output From the iostat n Command

For Each ... 

Field Name 

Description 

Terminal  

 

 

 

tin

Number of characters in the terminal input queue 

 

tout

Number of characters in the terminal output queue 

Disk  

 

 

 

bps

Blocks per second 

 

tps

Transactions per second 

 

serv

Average service time, in milliseconds 

CPU  

 

 

 

us

In user mode 

 

sy

In system mode 

 

wt

Waiting for I/O 

 

id

Idle 

Example--Displaying Disk Utilization Information

The following example shows disk statistics gathered every five seconds.


$ iostat 5
tty        sd0           sd6           nfs1         nfs49           cpu
tin tout kps tps serv  kps tps serv  kps tps serv  kps tps serv  us sy wt id
 0    0   1   0   49    0   0    0    0   0    0    0   0   15   0  0  0 100
 0   47   0   0    0    0   0    0    0   0    0    0   0    0   0  0  0 100
 0   16   0   0    0    0   0    0    0   0    0    0   0    0   0  0  0 100
 0   16   0   0    0    0   0    0    0   0    0    0   0    0   0  0  0 100
 0   16  44   6  132    0   0    0    0   0    0    0   0    0   0  0  1 99
 0   16   0   0    0    0   0    0    0   0    0    0   0    0   0  0  0 100
 0   16   0   0    0    0   0    0    0   0    0    0   0    0   0  0  0 100
 0   16   0   0    0    0   0    0    0   0    0    0   0    0   0  0  0 100
 0   16   0   0    0    0   0    0    0   0    0    0   0    0   0  0  0 100
 0   16   0   0    0    0   0    0    0   0    0    0   0    0   0  0  0 100
 0   16   3   1   23    0   0    0    0   0    0    0   0    0   0  0  1 99
 0   16   0   0    0    0   0    0    0   0    0    0   0    0   0  0  0 100
 0   16   0   0    0    0   0    0    0   0    0    0   0    0   0  0  0 100
 0   16   0   0    0    0   0    0    0   0    0    0   0    0   0  0  0 100

How to Display Extended Disk Statistics (iostat -xtc)

Run iostat -xtc to get extended disk statistics.


$ iostat -xtc
                  extended device statistics                      tty         cpu
device       r/s    w/s   kr/s   kw/s wait actv  svc_t  %w  %b  tin tout  us sy wt id
fd0          0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0    0    0   0  0  0 100
sd0          0.0    0.0    0.4    0.4  0.0  0.0   49.5   0   0 
sd6          0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0 
nfs1         0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0 
nfs49        0.0    0.0    0.0    0.0  0.0  0.0   15.1   0   0 
nfs53        0.0    0.0    0.4    0.0  0.0  0.0   24.5   0   0 
nfs54        0.0    0.0    0.0    0.0  0.0  0.0    6.3   0   0 
nfs55        0.0    0.0    0.0    0.0  0.0  0.0    4.9   0   0 

This command displays a line of output for each disk. The output fields are described in the table below.

Table 36-5 Output From the iostat -xtc Command

Field Name 

Description 

r/s

Reads per second 

w/s

Writes per second 

Kr/s

Kbytes read per second 

Kw/s

Kbytes written per second 

wait

Average number of transactions waiting for service (queue length) 

actv

Average number of transactions actively being serviced 

svc_t

Average service time, in milliseconds 

%w

Percentage of time the queue is not empty 

%b

Percentage of time the disk is busy  

Displaying Disk Usage Statistics (df)

Use the df command to show the amount of free disk space on each mounted disk. The usable disk space reported by df reflects only 90% of full capacity, as the reporting statistics leave a 10% head room above the total available space. This head room normally stays empty for better performance.

The percentage of disk space actually reported by df is used space divided by usable space.

If the file system is above 90% capacity, transfer files to a disk that is not as full by using cp, or to a tape by using tar or cpio; or remove the files.

For a detailed description of this command, refer to the df(1M) man page.

How to Display File System Information (df)

Use the df -k command to display file system information in Kbytes.


$ df -k
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0t3d0s0     192807   40231  133296    24%    / 

The table below describes the df -k command output.

Table 36-6 Output From the df -k Command

Field Name 

Description 

kbytes

Total size of usable space in the file system 

used

Amount of space used 

avail

Amount of space available for use 

capacity

Amount of space used, as a percent of the total capacity 

mounted on

Mount point 

Example--Displaying File System Information

The following example shows output of the df -k command.


$ df -k
/dev/dsk/c0t0d0s0     192807   49043  124484    29%    /
/dev/dsk/c0t0d0s6    1190551  680444  450580    61%    /usr
/proc                      0       0       0     0%    /proc
fd                         0       0       0     0%    /dev/fd
mnttab                     0       0       0     0%    /etc/mnttab
swap                  198056       0  198056     0%    /var/run
swap                  198064       8  198056     1%    /tmp
/dev/dsk/c0t0d0s5     192807    2031  171496     2%    /opt
/dev/dsk/c0t0d0s7     217191       9  195463     1%    /export/home
venus:/usr/dist    20612581 13237316 6963015    66%    /usr/dist

Monitoring System Activities (sar)

Use the sar command to:

For a detailed description of this command, refer to sar(1).

How to Check File Access (sar -a)

Display file access operation statistics with the sar -a command.


$ sar -a
SunOS venus 5.8 Generic sun4u    09/07/99

00:00:00  iget/s namei/s dirbk/s
01:00:00       0       0       0
02:00:02       0       0       0
03:00:00       0       1       0
04:00:00       0       0       0
05:00:01       0       0       0
06:00:00       0       0       0

Average        0       1       0

The operating system routines reported are described in the following table.

Table 36-7 Output from the sar -a Command

Field Name 

Description 

iget/s

The number of requests made for inodes that were not in the directory name lookup cache (dnlc).

namei/s

This is the number of file system path searches per second. If namei does not find a directory name in the dnlc, it calls iget to get the inode for either a file or directory. Hence, most igets are the result of dnlc misses.

dirbk/s

This is the number of directory block reads issued per second. 

The larger the values reported, the more time the kernel is spending to access user files. The amount of time reflects how heavily programs and applications are using the file systems. The -a option is helpful for viewing how disk-dependent an application is.

How to Check Buffer Activity (sar -b)

Display buffer activity statistics with the sar -b command.

The buffer is used to cache metadata, which includes inodes, cylinder group blocks, and indirect blocks.


$ sar -b
00:00:00 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
01:00:00       0       0     100       0       0      55       0       0

The buffer activities displayed by the -b option are described in the table below. The most important entries are the cache hit ratios %rcache and %wcache, which measure the effectiveness of system buffering. If %rcache falls below 90, or if %wcache falls below 65, it may be possible to improve performance by increasing the buffer space.

Table 36-8 Output from the sar -b Command

Field Name 

Description 

bread/s

Average number of reads per second submitted to the buffer cache from the disk 

lread/s

Average number of logical reads per second from the buffer cache 

%rcache

Fraction of logical reads found in the buffer cache (100% minus the ratio of bread/s to lread/s)

bwrit/s

Average number of physical blocks (512 blocks) written from the buffer cache to disk, per second 

lwrite/s

Average number of logical writes to the buffer cache, per second 

%wcache

Fraction of logical writes found in the buffer cache(100% minus the ratio of bwrit/s to lwrit/s)

pread/s

Average number of physical reads, per second, using character device interfaces 

pwrit/s

Average number of physical write requests, per second, using character device interfaces 

Example--Checking Buffer Activity

The following abbreviated example of sar -b output shows that the %rcache and %wcache buffers are not causing any slowdowns, because all the data is within acceptable limits.


$ sar -b
SunOS venus 5.8 Generic sun4u    09/07/99

00:00:00 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
01:00:00       0       0     100       0       0      55       0       0
02:00:02       0       0     100       0       0      55       0       0
03:00:00       0       0     100       0       0      72       0       0
04:00:00       0       0     100       0       0      56       0       0
05:00:01       0       0     100       0       0      55       0       0
06:00:00       0       0     100       0       0      55       0       0

Average        0       0      94       0       0      64       0       0

How to Check System Call Statistics (sar -c)

Display system call statistics by using the sar -c command.


$ sar -c
00:00:00 scall/s sread/s swrit/s  fork/s  exec/s rchar/s wchar/s
01:00:00      38       2       2    0.00    0.00     149     120

The table below describes the following system call categories reported by the -c option. Typically, reads and writes account for about half of the total system calls, although the percentage varies greatly with the activities that are being performed by the system.

Table 36-9 Output from the sar -c Command

Field Name 

Description 

scall/s

All types of system calls per second (generally about 30 per second on a busy four- to six-user system) 

sread/s

read system calls per second

swrit/s

write system calls per second

fork/s

fork system calls per second (about 0.5 per second on a four- to six-user system); this number will increase if shell scripts are running

exec/d

exec system calls per second; if exec/s divided by fork/s is greater than three, look for inefficient PATH variables

rchar/s

Characters (bytes) transferred by read system calls per second

wchar/s

Characters (bytes) transferred by write system calls per second

Example--Checking System Call Statistics

The following abbreviated example shows output from the sar -c command.


$ sar -c
SunOS venus 5.8 Generic sun4u    09/07/99

00:00:00 scall/s sread/s swrit/s  fork/s  exec/s rchar/s wchar/s
01:00:00      38       2       2    0.00    0.00     149     120
02:00:02      38       2       2    0.00    0.00     149     120
03:00:00      42       2       2    0.05    0.05     218     147
04:00:00      39       2       2    0.01    0.00     155     123
05:00:01      38       2       2    0.00    0.00     150     120
06:00:00      38       2       2    0.01    0.00     149     120

Average       50       4       3    0.02    0.02     532     238

How to Check Disk Activity (sar -d)

Display disk activity statistics with the sar -d command.


$ sar -d
00:00:00   device        %busy   avque   r+w/s  blks/s  avwait  avserv

01:00:00   fd0               0     0.0       0       0     0.0     0.0

The table below describes the disk devices activities reported by the -d option. Note that queue lengths and wait times are measured when there is something in the queue. If %busy is small, large queues and service times probably represent the periodic efforts by the system to ensure that altered blocks are written to the disk in a timely fashion.

Table 36-10 Output from the sar -d Command

Field Name 

Description 

device

Name of the disk device being monitored 

%busy

Percentage of time the device spent servicing a transfer request 

avque

The sum of the average wait time plus the average service time 

r+w/s

Number of read and write transfers to the device per second 

blks/s

Number of 512-byte blocks transferred to the device per second 

avwait

Average time, in milliseconds, that transfer requests wait idly in the queue (measured only when the queue is occupied) 

avserv

Average time, in milliseconds, for a transfer request to be completed by the device (for disks, this includes seek, rotational latency, and data transfer times) 

Examples--Checking Disk Activity

This abbreviated example illustrates the sar -d output.


$ sar -d
SunOS venus 5.8 Generic sun4u    09/07/99

00:00:00   device        %busy   avque   r+w/s  blks/s  avwait  avserv

01:00:00   fd0               0     0.0       0       0     0.0     0.0
           nfs1              0     0.0       0       0     0.0     0.0
           sd0               0     0.0       0       0     0.0    39.6
           sd0,a             0     0.0       0       0     0.0    39.6
           sd0,b             0     0.0       0       0     0.0     0.0
           sd0,c             0     0.0       0       0     0.0     0.0
           sd0,f             0     0.0       0       0     0.0     0.0
           sd0,g             0     0.0       0       0     0.0     0.0
           sd0,h             0     0.0       0       0     0.0     0.0
           sd6               0     0.0       0       0     0.0     0.0

How to Check Page-Out and Memory (sar -g)

Use the sar -g option reports page-out and memory freeing activities (in averages).


$ sar -g
00:00:00  pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf
01:00:00     0.00     0.00     0.00     0.00     0.00

The output displayed by sar -g is a good indicator of whether more memory may be needed. Use the ps -elf command to show the number of cycles used by the page daemon. A high number of cycles, combined with high values for pgfree/s and pgscan/s indicates a memory shortage.

The sar -g command also shows whether inodes are being recycled too quickly, causing a loss of reusable pages.

Output from the -g option is described in the following table.

Table 36-11 Output From the sar -g Command

Field Name 

Description 

pgout/s

The number of page-out requests per second. 

ppgout/s

The actual number of pages that are paged-out, per second. (A single page-out request may involve paging-out multiple pages.) 

pgfree/s

The number of pages, per second, that are placed on the free list. 

pgscan/s

The number of pages, per second, scanned by the page daemon. If this value is high, the page daemon is spending a lot of time checking for free memory. This implies that more memory may be needed.  

%ufs_ipf

The percentage of ufs inodes taken off the free list by iget that had reusable pages associated with them. These pages are flushed and cannot be reclaimed by processes. Thus, this is the percentage of igets with page flushes. A high value indicates that the free list of inodes is page-bound and the number of ufs inodes may need to be increased.

Example--Checking Page-Out and Memory

The following abbreviated example shows output from the sar -g command.


$ sar -g
SunOS venus 5.8 Generic sun4u    09/07/99

00:00:00  pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf
01:00:00     0.00     0.00     0.00     0.00     0.00
02:00:02     0.00     0.00     0.00     0.00     0.00
03:00:00     0.00     0.01     0.01     0.00     0.00
04:00:00     0.00     0.00     0.00     0.00     0.00
05:00:01     0.00     0.00     0.00     0.00     0.00
06:00:00     0.00     0.00     0.00     0.00     0.00

Average      0.01     0.12     0.21     0.66     0.00

How to Check Kernel Memory Allocation (sar -k)

Use the sar -k command to report on the following activities of the Kernel Memory Allocator (KMA).

The KMA allows a kernel subsystem to allocate and free memory as needed. Rather than statically allocating the maximum amount of memory it is expected to require under peak load, the KMA divides requests for memory into three categories: small (less than 256 bytes), large (512 to 4 Kbytes), and oversized (greater than 4 Kbytes). It keeps two pools of memory to satisfy small and large requests. The oversized requests are satisfied by allocating memory from the system page allocator.

If you are investigating a system that is being used to write drivers or STREAMS that use KMA resources, then sar -k will likely prove useful. Otherwise, you will probably not need the information it provides. Any driver or module that uses KMA resources, but does not specifically return the resources before it exits, can create a memory leak. A memory leak causes the amount of memory allocated by KMA to increase over time. Thus, if the alloc fields of sar -k increase steadily over time, there may be a memory leak. Another indication of a memory leak is failed requests. If this occurs, a memory leak has probably caused KMA to be unable to reserve and allocate memory.

If it appears that a memory leak has occurred, you should check any drivers or STREAMS that may have requested memory from KMA and not returned it.


$ sar -k
00:00:00 sml_mem   alloc  fail  lg_mem   alloc   fail  ovsz_alloc  fail
01:00:00 2523136 1866512     0 18939904 14762364    0      360448     0
02:00:02 2523136 1861724     0 18939904 14778748    0      360448     0

Output from the -k option is described in the table below.

Table 36-12 Output From the sar -k Command

Field Name 

Description 

sml_mem

The amount of memory, in bytes, that the KMA has available in the small memory request pool (a small request is less than 256 bytes) 

alloc

The amount of memory, in bytes, that the KMA has allocated from its small memory request pool to small memory requests 

fail

The number of requests for small amounts of memory that failed 

lg_mem

The amount of memory, in bytes, that the KMA has available in the large memory request pool (a large request is from 512 bytes to 4 Kbytes) 

alloc

The amount of memory, in bytes, that the KMA has allocated from its large memory request pool to large memory requests 

fail

The number of failed requests for large amounts of memory 

ovsz_alloc

The amount of memory allocated for oversized requests (those greater than 4 Kbytes); these requests are satisfied by the page allocator--thus, there is no pool 

fail

The number of failed requests for oversized amounts of memory 

Example--Checking Kernel Memory Allocation (sar)

The following is an abbreviated example of sar -k output.


$ sar -k
SunOS venus 5.8 Generic sun4u    09/07/99

00:00:00 sml_mem   alloc  fail  lg_mem   alloc  fail  ovsz_alloc  fail
01:00:00 2523136 1866512     0 18939904 14762364     0      360448     0
02:00:02 2523136 1861724     0 18939904 14778748     0      360448     0
03:00:00 2523136 1865664     0 18939904 14745884     0      360448     0
04:00:00 2523136 1867692     0 18939904 14746616     0      360448     0
05:00:01 2523136 1867208     0 18939904 14763700     0      360448     0
06:00:00 2523136 1867772     0 18939904 14779444     0      360448     0

Average  2724096 1791806     0 20089344 15434591     0      360448     0

How to Check Interprocess Communication (sar -m)

Use the sar -m command to report interprocess communication activities.


$ sar -m 
00:00:00   msg/s  sema/s
01:00:00    0.00    0.00

These figures will usually be zero (0.00), unless you are running applications that use messages or semaphores.

The output from the -m option is described in the table below.

Table 36-13 Output From the sar -m Command

Field Name 

Description 

msg/s

The number of message operations (sends and receives) per second. 

sema/s

The number of semaphore operations per second. 

Example--Checking Interprocess Communication

The following abbreviated example shows output from the sar -m command.


$ sar -m
SunOS venus 5.8 Generic sun4u    09/07/99

00:00:00   msg/s  sema/s
01:00:00    0.00    0.00
02:00:02    0.00    0.00
03:00:00    0.00    0.00
04:00:00    0.00    0.00
05:00:01    0.00    0.00
06:00:00    0.00    0.00

Average     0.00    0.00

How to Check Page-In Activity (sar -p)

Use the sar -p command to report page-in activity which includes protection and translation faults.


$ sar -p
00:00:00  atch/s  pgin/s ppgin/s  pflt/s  vflt/s slock/s
01:00:00    0.07    0.00    0.00    0.21    0.39    0.00

The reported statistics from the -p option are described in the table below.

Table 36-14 Output from the sar -p Command

Field Name 

Description 

atch/s

The number of page faults, per second, that are satisfied by reclaiming a page currently in memory (attaches per second). Instances of this include reclaiming an invalid page from the free list and sharing a page of text currently being used by another process (for example, two or more processes accessing the same program text). 

pgin/s

The number of times, per second, that file systems receive page-in requests. 

ppgin/s

The number of pages paged in, per second. A single page-in request, such as a soft-lock request (see slock/s), or a large block size, may involve paging-in multiple pages.

pflt/s

The number of page faults from protection errors. Instances of protection faults are illegal access to a page and "copy-on-writes." Generally, this number consists primarily of "copy-on-writes." 

vflt/s

The number of address translation page faults, per second. These are known as validity faults, and occur when a valid process table entry does not exist for a given virtual address. 

slock/s

The number of faults, per second, caused by software lock requests requiring physical I/O. An example of the occurrence of a soft-lock request is the transfer of data from a disk to memory. The system locks the page that is to receive the data, so that it cannot be claimed and used by another process.  

Example--Checking Page-In Activity

The following abbreviated example shows output from sar -p.


$ sar -p
SunOS venus 5.8 Generic sun4u    09/07/99

00:00:00  atch/s  pgin/s ppgin/s  pflt/s  vflt/s slock/s
01:00:00    0.07    0.00    0.00    0.21    0.39    0.00
02:00:02    0.07    0.00    0.00    0.21    0.39    0.00
03:00:00    0.32    0.00    0.00    1.10    2.48    0.00
04:00:00    0.09    0.00    0.00    0.32    0.57    0.00
05:00:01    0.07    0.00    0.00    0.21    0.39    0.00
06:00:00    0.07    0.00    0.00    0.21    0.39    0.00

Average     0.26    0.20    0.30    0.92    1.78    0.00

How to Check Queue Activity (sar -q)

Use the sar -q command to report the average queue length while the queue is occupied, and the percentage of time that the queue is occupied.


$ sar -q
00:00:00 runq-sz %runocc swpq-sz %swpocc
01:00:00

Note -

The number of LWPs swapped out may greater than zero even if the system has an abundance of free memory. This happens when a sleeping LWP is swapped out and has not been awakened (for example, a process or LWP sleeping, waiting for the keyboard or mouse input).


Output from the -q option is described in the table below.

Table 36-15 Output From the sar -q Command

Field Name 

Description 

runq-sz

The number of kernel threads in memory waiting for a CPU to run. Typically, this value should be less than 2. Consistently higher values mean that the system may be CPU-bound. 

%runocc

The percentage of time the dispatch queues are occupied. 

swpq-sz

The average number of swapped out LWPs. 

%swpocc

The percentage of time LWPs are swapped out. 

Example--Checking Queue Activity

The following abbreviated example shows output from the sar -q command. If %runocc is high (greater than 90 percent) and runq-sz is greater than 2, the CPU is heavily loaded and response is degraded. In this case, additional CPU capacity may be required to obtain acceptable system response.


$ sar -q
SunOS venus 5.8 Generic sun4u    09/07/99

00:00:00 runq-sz %runocc swpq-sz %swpocc
01:00:00                                
02:00:02                                
03:00:00     1.0       0                
04:00:00                                
05:00:01     1.0       0                
06:00:00

Average      1.3       0  

How to Check Unused Memory (sar -r)

Use the sar -r command to report the number of memory pages and swap-file disk blocks that are currently unused.


$ sar -r
00:00:00 freemem freeswap
01:00:00    2135   401922

Output from the -r option is described in the table below.

Table 36-16 Output From the sar -r Command

Field Name 

Description 

freemem

The average number of memory pages available to user processes over the intervals sampled by the command. Page size is machine-dependent. 

freeswap

The number of 512-byte disk blocks available for page swapping. 

Example--Checking Unused Memory

The following example shows output from the sar -r command.


$ sar -r
SunOS venus 5.8 Generic sun4u    09/07/99

00:00:00 freemem freeswap
01:00:00    2135   401922
02:00:02    2137   401949
03:00:00    2137   402006
04:00:00    2139   401923
05:00:01    2138   402033
06:00:00    2137   401919

Average     2500   399914

How to Check CPU Utilization (sar -u)

Display CPU utilization with the sar -u command.


$ sar -u
00:00:00    %usr    %sys    %wio   %idle
01:00:00       0       0       0     100

(The sar command without any options is equivalent to sar -u.) At any given moment, the processor is either busy or idle. When busy, the processor is in either user or system mode. When idle, the processor is either waiting for I/O completion or "sitting still" with no work to do.

Output from the -u option is described in the table below.

Table 36-17 Output From the sar -u Command

Field Name 

Description 

%sys

Lists the percentage of time that the processor is in system mode 

%user

Lists the percentage of time that the processor is in user mode 

%wio

Lists the percentage of time the processor is idle and waiting for I/O completion 

%idle

Lists the percentage of time the processor is idle and is not waiting for I/O 

A high %wio generally means a disk slowdown has occurred.

Example--Checking CPU Utilization

The following example shows output from the sar -u command.


$ sar -u
SunOS venus 5.8 Generic sun4u    09/07/99

00:00:00    %usr    %sys    %wio   %idle
01:00:00       0       0       0     100
02:00:02       0       0       0     100
03:00:00       0       0       0     100
04:00:00       0       0       0     100
05:00:01       0       0       0     100
06:00:00       0       0       0     100
07:00:00       0       0       0     100
08:00:01       0       0       0     100
08:20:00       0       0       0     100
08:40:00       0       0       0     100
09:00:00       0       0       0     100
09:20:00       0       0       0     100
09:40:00       0       0       0     100
10:00:00       0       0       0     100
10:20:00       0       0       0     100
10:40:01       0       0       0     100
11:00:00       5       2      10      82

Average        0       0       0     100

How to Check System Table Status (sar -v)

Use the sar -v command to report the status of the process table, inode table, file table, and shared memory record table.


$ sar -v
00:00:00  proc-sz    ov  inod-sz    ov  file-sz    ov   lock-sz
01:00:00   43/922     0 2984/4236    0  322/322     0    0/0   

Output from the -v option is described in the table below.

Table 36-18 Output From the sar -v Command

Field Name 

Description 

proc-sz

The number of process entries (proc structs) currently being used, or allocated in the kernel.

inod-sz

The total number of inodes in memory verses the maximum number of inodes allocated in the kernel. This is not a strict high water mark; it can overflow. 

file-sz

The size of the open system file table. The sz is given as 0, since space is allocated dynamically for the file table.

ov

The number of shared memory record table entries currently being used or allocated in the kernel. The sz is given as 0 because space is allocated dynamically for the shared memory record table.

lock-sz

The number of shared memory record table entries currently being used or allocated in the kernel. The sz is given as 0 because space is allocated dynamically for the shared memory record table.

Example--Checking System Table Status

The following abbreviated example shows output from the sar -v command. This example shows that all tables are large enough to have no overflows. These tables are all dynamically allocated based on the amount of physical memory.


$ sar -v
SunOS venus 5.8 Generic sun4u    09/07/99

00:00:00  proc-sz    ov  inod-sz    ov  file-sz    ov   lock-sz
01:00:00   43/922     0 2984/4236    0  322/322     0    0/0   
02:00:02   43/922     0 2984/4236    0  322/322     0    0/0   
03:00:00   43/922     0 2986/4236    0  323/323     0    0/0   
04:00:00   43/922     0 2987/4236    0  322/322     0    0/0   
05:00:01   43/922     0 2987/4236    0  322/322     0    0/0   
06:00:00   43/922     0 2987/4236    0  322/322     0    0/0   

How to Check Swap Activity (sar -w)

Use the sar -w command to report swapping and switching activity.


$ sar -w
00:00:00 swpin/s bswin/s swpot/s bswot/s pswch/s
01:00:00    0.00     0.0    0.00     0.0      22

Target values and observations are described in the table below.

Table 36-19 Output From the sar -w Command

Field Name 

Description 

swpin/s

The number of LWP transfers into memory per second. 

bswin/s

The average number of processes swapped out of memory per second. If the number is greater than 1, you may need to increase memory. 

swpot/s

The average number of processes swapped out of memory per second. If the number is greater than 1, you may need to increase memory. 

bswot/s

The number of blocks transferred for swap-outs per second. 

pswch/s

The number of kernel thread switches per second. 


Note -

All process swap-ins include process initialization.


Example--Checking Swap Activity

The following example shows output from the sar -w command.


$ sar -w
SunOS venus 5.8 Generic sun4u    09/07/99

00:00:00 swpin/s bswin/s swpot/s bswot/s pswch/s
01:00:00    0.00     0.0    0.00     0.0      22
02:00:02    0.00     0.0    0.00     0.0      22
03:00:00    0.00     0.0    0.00     0.0      22
04:00:00    0.00     0.0    0.00     0.0      22
05:00:01    0.00     0.0    0.00     0.0      22
06:00:00    0.00     0.0    0.00     0.0      22
07:00:00    0.00     0.0    0.00     0.0      22
08:00:01    0.00     0.0    0.00     0.0      22
08:20:00    0.00     0.0    0.00     0.0      22
08:40:00    0.00     0.0    0.00     0.0      22
09:00:00    0.00     0.0    0.00     0.0      22
09:20:00    0.00     0.0    0.00     0.0      22
09:40:00    0.00     0.0    0.00     0.0      22
10:00:00    0.00     0.0    0.00     0.0      22
10:20:00    0.00     0.0    0.00     0.0      22
10:40:01    0.00     0.0    0.00     0.0      23
11:00:00    0.00     0.0    0.00     0.0     144

Average     0.00     0.0    0.00     0.0      24

How to Check Terminal Activity (sar -y)

Use the sar -y command to monitor terminal device activities.


$ sar -y
00:00:00 rawch/s canch/s outch/s rcvin/s xmtin/s mdmin/s
01:00:00       0       0       0       0       0       0

If you have a lot of terminal I/O, you can use this report to determine if there are any bad lines. The activities recorded are defined in the table below.

Table 36-20 Output From the sar -y Command

Field Name 

Description 

rawch/s

Input characters (raw queue), per second 

canch/s

Input characters processed by canon (canonical queue) per second 

outch/s

Output characters (output queue) per second 

rcvin/s

Receiver hardware interrupts per second 

xmtin/s

Transmitter hardware interrupts per second 

mdmin/s

Modem interrupts per second 

The number of modem interrupts per second (mdmin/s) should be close to zero, and the receive and transmit interrupts per second (xmtin/s and rcvin/s) should be less than or equal to the number of incoming or outgoing characters, respectively. If this is not the case, check for bad lines.

Example--Checking Terminal Activity

The following abbreviated example shows output from the sar -y command.


$ sar -y
SunOS venus 5.8 Generic sun4u    09/07/99

00:00:00 rawch/s canch/s outch/s rcvin/s xmtin/s mdmin/s
01:00:00       0       0       0       0       0       0
02:00:02       0       0       0       0       0       0
03:00:00       0       0       0       0       0       0
04:00:00       0       0       0       0       0       0
05:00:01       0       0       0       0       0       0
06:00:00       0       0       0       0       0       0
07:00:00       0       0       0       0       0       0
08:00:01       0       0       0       0       0       0
08:20:00       0       0       0       0       0       0
08:40:00       0       0       0       0       0       0
09:00:00       0       0       0       0       0       0
09:20:00       0       0       0       0       0       0
09:40:00       0       0       0       0       0       0
10:00:00       0       0       0       0       0       0
10:20:00       0       0       0       0       0       0
10:40:01       0       0      20       0       0       0

Average        0       0       3       0       0       0

How to Check Overall System Performance (sar -A)

Use the sar -A command to display a view of overall system performance.

This provides a more global perspective. If data from more than one time segment is shown, the report includes averages.

Collecting System Activity Data Automatically (sar)

Three commands are involved in automatic system activity data collection: sadc, sa1, and sa2.

The sadc data collection utility periodically collects data on system activity and saves it in a file in binary format--one file for each 24-hour period. You can set up sadc to run periodically (usually once each hour), and whenever the system boots to multiuser mode. The data files are placed in the directory /usr/adm/sa. Each file is named sadd, where dd is the current date. The format of the command is as follows:


/usr/lib/sa/sadc [t n] [ofile]

The command samples n times with an interval of t seconds (t should be greater than 5 seconds) between samples. It then writes, in binary format, to the file ofile, or to standard output. If t and n are omitted, a special file is written once.

Running sadc When Booting

The sadc command should be run at system boot time in order to record the statistics from when the counters are reset to zero. To make sure that sadc is run at boot time, the /etc/init.d/perf file must contain a command line that writes a record to the daily data file.

The command entry has the following format:


su sys -c "/usr/lib/sa/sadc /usr/adm/sa/sa`date +5d`"

Running sadc Periodically With sa1

To generate periodic records, you need to run sadc regularly. The simplest way to do this is by putting a line into the /var/spool/cron/sys file, which calls the shell script, sa1. This script invokes sadc and writes to the daily data files, /var/adm/sa/sadd. It has the following format:


/usr/lib/sa/sa1 [t n]

The arguments t and n cause records to be written n times at an interval of t seconds. If these arguments are omitted, the records are written only one time.

Producing Reports With sa2

Another shell script, sa2, produces reports rather than binary data files. The sa2 command invokes the sar command and writes the ASCII output to a report file.

Collecting System Activity Data (sar)

The sar command can be used either to gather system activity data itself or to report what has been collected in the daily activity files created by sadc.

The sar command has the following formats:


sar [-aAbcdgkmpqruvwy] [-o file] t [n] 

sar [-aAbcdgkmpqruvwy] [-s time] [-e time] [-i sec] [-f file]

The sar command below samples cumulative activity counters in the operating system every t seconds, n times. (t should be 5 seconds or greater; otherwise, the command itself may affect the sample.) You must specify a time interval between which to take the samples; otherwise, the command operates according to the second format. The default value of n is 1. The following example takes two samples separated by 10 seconds. If the -o option is specified, samples are saved in file in binary format.


$ sar -u 10 2

Other important information about the sar command:

The table below lists the sar options and their actions.

Table 36-21 Options for sar Command

Option 

Actions 

-a

Checks file access operations 

-b

Checks buffer activity  

-c

Checks system calls  

-d

Checks activity for each block device 

-g

Checks page-out and memory freeing  

-k

Checks kernel memory allocation  

-m

Checks interprocess communication  

-p

Checks swap and dispatch activity  

-q

Checks queue activity  

-r

Checks unused memory  

-u

Checks CPU utilization 

-nv

Checks system table status  

-w

Checks swapping and switching volume  

-y

Checks terminal activity  

-A

Reports overall system performance (same as entering all options) 

If no option is used, it is equivalent to calling the command with the -u option.

How to Set Up Automatic Data Collection

  1. Become superuser.

  2. Edit the /etc/init.d/perf file and uncomment all lines:

    This version of the sadc command writes a special record that marks the time when the counters are reset to zero (boot time). The sadc output is put into the file sadd (where dd is the current date), which acts as the daily system activity record.

  3. Edit the /var/spool/cron/crontabs/sys file (the system crontab file) and uncomment the following lines:


    # 0 * * * 0-6 /usr/lib/sa/sa1
    # 20,40 8-17 * * 1-5 /usr/lib/sa/sa1
    # 5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A

    The first entry writes a record to /var/adm/sa/sadd on the hour, every hour, seven days a week.

    The second entry writes a record to /var/adm/sa/sadd twice each hour during peak working hours: at 20 minutes and 40 minutes past the hour, from 8 a.m. to 5 p.m., Monday through Friday.

    Thus, these two crontab entries cause a record to be written to /var/adm/sa/sadd every 20 minutes from 8 a.m. to 5 p.m., Monday through Friday, and every hour on the hour otherwise. You can change these defaults to meet your needs.