Part XV Managing System Performance

This part provides instructions for managing system performance. This part contains these chapters.

Chapter 62 System Performance (Overview)

Getting good performance from a computer or network is an important part of system administration. This chapter is an overview of some of the factors that contribute to maintaining and managing the performance of the computer systems in your care.

This is a list of the overview information in this chapter.

• "What's New in System Performance"

• "Where to Find System Performance Tasks"

• "System Performance and System Resources"

• "Processes and System Performance"

• "Disk I/O and System Performance"

• "Memory and System Performance"

• "Kernel Parameters and System Performance"

• "About Monitoring Performance"

What's New in System Performance

This section describes new system performance features in the Solaris 2.6 release.

Scheduler Activation

Scheduler activations provide kernel scheduling support for applications with particular scheduling needs, such as database and multithreaded applications. Multithreaded support changes for scheduler activation are implemented as a private interface between the kernel and the libthread library, without changing the libthread interface. Additionally, applications may give scheduling hints to the kernel to improve performance. See libsched(3X) for more information.

The psrset Command

Another new feature allows a group of processors to be allocated for the exclusive use of one or more applications. The /usr/sbin/psrset command gives a system administrator control over the creation and management of processor sets. See psrset(1M) for more information.

UFS Direct Input/Output (I/O)

Direct I/O is intended to boost bulk I/O operations. Bulk I/O operations use large buffer sizes to transfer large files (files larger than physical memory).

An example of a bulk I/O operation is downloading satellite data, which writes large amounts of data to a file. Direct I/O data is read or written into memory without using the overhead of the operating system's page caching mechanism.

There is a potential penalty on direct I/O startup. If a file requested for I/O is already mapped by another application, the pages will have to be flushed out of memory before the direct I/O operation can begin.

See directio(3C) for more information.

Direct I/O can also be enabled on a file system by using the forcedirectio option to the mount command. Enabling direct I/O is a performance benefit only when a file system is transferring large amounts of sequential data.

When a file system is mounted with this option, data is transferred directly between a user's address space and the disk. When forced direct I/O is not enabled for a file system, data transferred between a user's address space and the disk is first buffered in the kernel address space.

The default behavior is no forced direct I/O on a UFS file system. See mount_ufs for more information.

How to Enable Forced Direct I/O on a UFS File System

1. Become superuser.

2. Mount a file system with the forcedirectio mount option.

   # mount -F ufs -o forcedirectio /dev/dsk/c0t3d0s7 /datab

3. Verify the mounted file system has forced direct I/O enabled.

   # mount . . . /export/home on /dev/dsk/c0t3d0s7 forcedirectio/setuid/read/write/largefiles on Mon May 12 13:47:55 1997

Enhancements to the /proc File System and Watchpoints

The previous flat /proc file system has been restructured into a directory hierarchy that contains additional sub-directories for state information and control functions.

It also provides a watchpoint facility that is used to remap read/write permissions on the individual pages of a process's address space. This facility has no restrictions and is MT-safe.

The new /proc file structure provides complete binary compatibility with the old /proc interface except that the new watchpoint facility cannot be used with the old interface.

Debugging tools have been modified to use /proc's new watchpoint facility, which means the entire watchpoint process is faster.

The following restrictions have been removed when setting watchpoints using the dbx debugging tool:

• Setting watchpoints on local variables on the stack due to SPARC register windows

• Setting watchpoints on multi-threaded processes

See proc(4), core(4), adb(4) for more information.

Where to Find System Performance Tasks

Use these references to find step-by-step instructions for monitoring system performance.

• Chapter 63, Managing Processes (Tasks)

• Chapter 64, Monitoring Performance (Tasks)

• Chapter 65, Monitoring Network Performance (Tasks)

• Chapter 66, Tuning Kernel Parameters (Tasks)

System Performance and System Resources

The performance of a computer system depends upon how the system uses and allocates its resources. It is important to monitor your system's performance on a regularly so that you know how it behaves under normal conditions. You should have a good idea of what to expect, and be able to recognize a problem when it occurs.

System resources that affect performance include:

• Central processing unit (CPU) - The CPU processes instructions, fetching instructions from memory and executing them.

• Input/output (I/O) devices - I/O devices transfer information into and out of the computer. Such a device could be a terminal and keyboard, a disk drive, or a printer.

• Memory - Physical (or main) memory is the amount of memory (RAM) on the system.

Chapter 64, Monitoring Performance (Tasks) describes the tools that display statistics about the activity and the performance of the computer system.

Other Sources of Information

Performance is a broad subject that can't be adequately covered in these chapters. There are several books available that cover various aspects of improving performance and tuning your system or network. Three useful books are:

• Sun Performance and Tuning: SPARC and Solaris, by Adrian Cockcroft, SunSoft Press/PRT Prentice Hall, ISBN 0-13-149642-3

• System Performance Tuning, by Mike Loukides, O'Reilly & Associates, Inc.

• Managing NFS and NIS, by Hal Stern, O'Reilly & Associates, Inc.

Processes and System Performance

Terms related to processes are described in Table 62-1.

A process can consist of multiple LWPs and multiple application threads. The kernel schedules a kernel-thread structure, which is the scheduling entity in the SunOS 5.x environment. Various process structures are described in Table 62-2.

Figure 62-1 illustrates the relationship of these structures.

Figure 62-1 Process Structures

Most process resources are accessible to all the threads in the process. Almost all process virtual memory is shared. A change in shared data by one thread is available to the other threads in the process.

Commands for Managing Processes

Table 62-3 describes commands for managing processes.

See Chapter 63, Managing Processes (Tasks) for more information about commands for managing processes.

In addition, process tools are available in /usr/proc/bin that display highly detailed information about the processes listed in /proc, also known as the process file system (PROCFS). Images of active processes are stored here by their process ID number.

The process tools are similar to some options of the ps command, except that the output provided by the tools is more detailed. In general, the process tools:

• Display more details about processes, such as fstat and fcntl information, working directories, and trees of parent and child processes

• Provide control over processes, allowing users to stop or resume them

The new /usr/proc/bin utilities are summarized in Table 62-4.

In these commands, pid is a process identification number. You can obtain this number by using the ps -ef command.

Chapter 63, Managing Processes (Tasks), describes how to use the process tool commands to perform selected system administration tasks, such as displaying details about processes, and starting and stopping them. A more detailed description of the process tools can be found in proc(1).

If a process becomes trapped in an endless loop, or if it takes too long to execute, you may want to stop (kill) the process. See Chapter 63, Managing Processes (Tasks), for more information about stopping processes using the kill command.

Process Scheduling Classes and Priority Levels

A process is allocated CPU time according to its scheduling class and its priority level. By default, the Solaris operating system has four process scheduling classes: real-time, system, timesharing and interactive.

• Real-time processes have the highest priority. This class includes processes that must respond to external events as they happen. For example, a process that collects data from a sensing device may need to process the data and respond immediately. In most cases, a real-time process requires a dedicated system. No other processes can be serviced while a real-time process has control of the CPU. By default, the range of priorities is 100-159.

• System processes have the middle priorities. This class is made up of those processes that are automatically run by the kernel, such as the swapper and the paging daemon. By default, the range of priorities is 60-99.

• Timesharing processes have the lowest priority. This class includes the standard UNIX processes. Normally, all user processes are timesharing processes. They are subject to a scheduling policy that attempts to distribute processing time fairly, giving interactive applications quick response time and maintaining good throughput for computations. By default, the range of priorities is 0-59.

• Interactive processes are introduced in the SunOS 5.4 environment. The priorities range from 0-59. All processes started under OpenWindows are placed in the interactive class and those processes with keyboard focus get higher priorities.

The scheduling priority determines the order in which processes will be run.

Real-time processes have fixed priorities. If a real-time process is ready to run, no system process or timesharing process can run.

System processes have fixed priorities that are established by the kernel when they are started. The processes in the system class are controlled by the kernel, and cannot be changed.

Timesharing and interactive processes are controlled by the scheduler, which dynamically assigns their priorities. You can manipulate the priorities of the processes within this class.

Disk I/O and System Performance

The disk is used to store data and instructions used by your computer system. You can examine how efficiently the system is accessing data on the disk by looking at the disk access activity and terminal activity. See Chapter 64, Monitoring Performance (Tasks) for a discussion of the iostat and sar commands, which report statistics on disk activity. Managing and allocating disk space and dividing your disk into slices are discussed in Chapter 21, Disk Management (Overview).

If the CPU spends much of its time waiting for I/O completions, there is a problem with disk slowdown. Some ways to prevent disk slowdowns are:

• Keep disk space with 10% free so file systems are not full. If a disk becomes full, back up and restore the file systems to prevent disk fragmentation. Consider purchasing products that resolve disk fragmentation.

• Organize the file system to minimize disk activity. If you have two disks, distribute the file system for a more balanced load. Using Sun's Solstice DiskSuite(TM) product provides more efficient disk usage.

• Add more memory. Additional memory reduces swapping and paging traffic, and allows an expanded buffer pool (reducing the number of user-level reads and writes that need to go out to disk).

• Add a disk and balance the most active file systems across the disks.

Memory and System Performance

Performance suffers when the programs running on the system require more physical memory than is available. When this happens, the operating system begins paging and swapping, which is costly in both disk and CPU overhead.

Paging involves moving pages that have not been recently referenced to a free list of available memory pages. Most of the kernel resides in main memory and is not pageable.

Swapping occurs if the page daemon cannot keep up with the demand for memory. The swapper will attempt to swap out sleeping or stopped lightweight processes (LWPs). If there are no sleeping or stopped LWPs, the swapper will swap out a runnable process. The swapper will swap LWPs back in based on their priority. It will attempt to swap in processes that are runnable.

Swap Space

Swap areas are really file systems used for swapping. Swap areas should be sized based on the requirements of your applications. Check with your vendor to identify application requirements.

Table 62-5 describes the formula used to size default swap areas by the Solaris 2.x installation program. These default swap sizes are a good place to start if you are not sure how to size your swap areas.

See Chapter 30, Configuring Additional Swap Space (Tasks) for information about managing swap space.

Buffer Resources

The buffer cache for read and write system calls uses a range of virtual addresses in the kernel address space. A page of data is mapped into the kernel address space and the amount of data requested by the process is then physically copied to the process' address space. The page is then unmapped in the kernel. The physical page will remain in memory until the page is freed up by the page daemon.

This means a few I/O-intensive processes can monopolize or force other processes out of main memory. To prevent monopolization of main memory, balance the running of I/O-intensive processes serially in a script or with the at command. Programmers can use mmap(2) and madvise(3) to ensure that their programs free memory when they are not using it.

Kernel Parameters and System Performance

Many basic parameters (or tables) within the kernel are calculated from the value of the maxusers parameter. Tables are allocated space dynamically. However, you can set maximums for these tables to ensure that applications won't take up large amounts of memory.

By default, maxusers is approximately set to the number of Mbytes of physical memory on the system. However, the system will never set maxusers higher than 1024. The maximum value of maxusers is 2048, which can be set by modifying the /etc/system file.

See Chapter 66, Tuning Kernel Parameters (Tasks), and system(3S) for details on kernel parameters.

In addition to maxusers, a number of kernel parameters are allocated dynamically based on the amount of physical memory on the system, as shown in Table 62-6 below.

Table 62-7 lists the default settings for kernel parameters affected by the value assigned to maxusers.

See Chapter 66, Tuning Kernel Parameters (Tasks), for a description of the kernel parameters and how to change the default values.

About Monitoring Performance

While your computer is running, counters in the operating system are incremented to keep track of various system activities. System activities that are tracked are:

• Central processing unit (CPU) utilization

• Buffer usage

• Disk and tape input/output (I/O) activity

• Terminal device activity

• System call activity

• Context switching

• File access

• Queue activity

• Kernel tables

• Interprocess communication

• Paging

• Free memory and swap space

• Kernel Memory Allocation (KMA)

Monitoring Tools

The Solaris 2.x system software provides several tools to help you keep track of how your system is performing. These include:

Chapter 63 Managing Processes (Tasks)

This chapter describes the procedures for managing system processes. This is a list of the step-by-step instructions in this chapter.

• "How to List Processes"

• "How to Display Information About Processes"

• "How to Control Processes"

• "How to Kill a Process"

• "How to Display Basic Information About Process Classes"

• "How to Display the Global Priority of a Process"

• "How to Designate a Process Priority"

• "How to Change Scheduling Parameters of a Timeshare Process"

• "How to Change the Class of a Process"

• "How to Change the Priority of a Process"

Displaying Information About Processes

This section describes commands used to manage process information.

The ps Command

The ps command enables you to check the status of active processes on a system, as well as display technical information about the processes. This data is useful for such administrative tasks as determining how to set process priorities, and how to kill processes that have hung or become inactive.

Depending on which options you use, ps reports the following information:

• Current status of the process

• Process ID

• Parent process ID

• User ID

• Scheduling class

• Priority

• Address of the process

• Memory used

• CPU time used

Table 63-1 describes some of the fields reported by the ps command. The fields displayed depend on which option you choose. See ps(1) for a description of all available options.

How to List Processes

To list all the processes being executed on a system, use the ps command.

$ ps [-ef]

Example--Listing Processes

The following example shows output from the ps command when no options are used.

$ ps
   PID TTY      TIME COMD
  1664 pts/4    0:06 csh
  2081 pts/4    0:00 ps

The following example shows output from ps -ef. This shows that the first process executed when the system boots is sched (the swapper) followed by the init process, pageout, and so on.

$ ps -ef
     UID   PID  PPID  C    STIME TTY     TIME CMD
    root     0     0  0   May 05 ?       0:04 sched
    root     1     0  0   May 05 ?      10:48 /etc/init -
    root     2     0  0   May 05 ?       0:00 pageout
    root     3     0  0   May 05 ?      43:21 fsflush
    root   238     1  0   May 05 ?       0:00 /usr/lib/saf/sac -t 300
    root   115     1  0   May 05 ?       0:10 /usr/sbin/rpcbind
    root   158     1  0   May 05 ?       0:00 /usr/lib/autofs/autom...
    root   134     1  0   May 05 ?       0:12 /usr/sbin/inetd -s
    root   107     1  0   May 05 ?      11:49 /usr/sbin/in.routed -q
    root   117     1  5   May 05 ?     899:32 /usr/sbin/keyserv
    root   125     1  0   May 05 ?       0:00 /usr/sbin/kerbd
    root   123     1  0   May 05 ?       4:17 /usr/sbin/nis_cachemgr
    root   137     1  0   May 05 ?       0:00 /usr/lib/nfs/statd
    root   139     1  0   May 05 ?       0:02 /usr/lib/nfs/lockd
    root   159     1 50   May 05 ?    8243:36 /usr/sbin/automount
    root   199   191  0   May 05 ?       0:00 lpNet
    root   162     1  0   May 05 ?       0:07 /usr/sbin/syslogd
    root   181     1  0   May 05 ?       0:03 /usr/sbin/nscd... 
    root   169     1  0   May 05 ?       5:09 /usr/sbin/cron
    root   191     1  0   May 05 ?       0:00 /usr/lib/lpsched
    root   210     1  0   May 05 ?       0:01 /usr/sbin/vold
    root   200     1  0   May 05 ?       0:08 /usr/lib/sendmail -bd -q1h
    root  4942     1  0   May 17 console 0:00 /usr/lib/saf/ttymon...
    root   208     1  0   May 05 ?       0:00 /usr/lib/utmpd
    root   241   238  0   May 05 ?       0:00 /usr/lib/saf/ttymon
    root  5748   134  0 17:09:49 ?       0:01 in.rlogind
    root  5750  5748  0 17:09:52 pts/0   0:00 -sh
    root  5770  5750  2 17:23:39 pts/0   0:00 ps -ef

Displaying Information About Processes (/proc Tools)

You can display detailed, technical information about active processes by using some of the process tool commands contained in /usr/proc/bin. Table 63-2 lists these process tools. Refer to proc(1)for more information.

To avoid typing long command names, add the process tool directory to your PATH variable. This enables you to run process tools by entering only the last part of each file name (for example, pwdx instead of /usr/proc/bin/pwdx).

How to Display Information About Processes

1. (Optional) Use output from the ps command to obtain the identification number of the process you want to display more information about.

   # ps -e | grep process

   process

   Name of the process you want to display more information about.

   The process identification number is in the first column of the output.

2. Use the appropriate /usr/bin/proc command to display the information you need.

   # /usr/proc/bin/pcommand pid

   pcommand	Process tool command you want to run. Table 63-2 lists these commands.
   pid	Identification number of a process.

Examples--Displaying Information About Processes

The following example shows how to use process tool commands to display more information about an lpsched process. First the /usr/proc/bin path is defined to avoid typing long process tool commands. Next, the identification number for lpsched is obtained. Finally, output from three process tool commands is shown.

 [Adds the /usr/proc/bin directory to
the PATH variable. ] # PATH=$PATH:/usr/proc/bin
# export PATH
 [Obtains the process identification number for lpsched. ] # ps -e | grep lpsched
   191 ?        0:00 /usr/lib/lpsched
 [Displays the current working directory for lpsched.] # pwdx 191
 
191:    /
 [	Displays the process tree containing lpsched. ] # ptree 191
 
183   /usr/lib/lpsched
 [Displays fstat and fcntl
information. ] # pfiles 191
210:    /usr/lib/lpsched
  Current rlimit: 1024 file descriptors
   0: S_IFIFO mode:0000 dev:165,0 ino:83 uid:0 gid:0 size:0
      O_RDWR
   1: S_IFIFO mode:0000 dev:165,0 ino:83 uid:0 gid:0 size:0
      O_RDWR
   3: S_IFCHR mode:0666 dev:32,24 ino:34307 uid:0 gid:3 rdev:21,0
      O_WRONLY FD_CLOEXEC
   4: S_IFDOOR mode:0444 dev:171,0 ino:4124226512 uid:0 gid:0 
      size:0
      O_RDONLY|O_LARGEFILE FD_CLOEXEC  door to nscd[200]
   5: S_IFREG mode:0664 dev:32,24 ino:311 uid:71 gid:8 size:0
      O_WRONLY

The following example shows output from the pwait command, which waits until a process terminates, then displays information about what happened. The following example shows output from the pwait command after a Command Tool window was exited.

$ ps -e | grep cmdtool
  273 console 0:01 cmdtool
  277 console 0:01 cmdtool
  281 console 0:01 cmdtool
$ pwait -v 281
281: terminated, wait status 0x0000
 

Controlling Processes (/proc Tools)

You can control some aspects of processes by using some of the process tools contained in /usr/proc/bin. Table 63-3 lists these process tools. Refer to proc(1) for detailed information about process tools.

To avoid typing long command names, add the process tool directory to your PATH variable. This allows you to run process tools by entering only the last part of each file name (for example, prun instead of /usr/proc/bin/prun).

How to Control Processes

1. (Optional) Use output from the ps command to obtain the identification number of the process you want to display more information about.

   # ps -e | grep process

   process

   Name of the process you want to display more information about.

   The process identification number is in the first column of the output.

2. Use the appropriate /usr/proc/bin command to control the process.

   # /usr/proc/bin/pcommand PID

   pcommand	Process tool command you want to run. Table 63-3 lists these commands.
   PID	Identification number of a process.

3. Verify the process status using the ps command.

   # ps | grep PID

Example--Controlling Processes

The following example shows how to use process tools to stop and restart Print Tool.

 [Adds the /usr/proc/bin directory to
the PATH variable.] # PATH=$PATH:/usr/proc/bin
# export PATH
 [Obtains the process identification number for Print Tool. ] # ps -e | grep print*
264 console 0:03 printtool
 [Stops the Print Tool process. ] # pstop 264
  [Restarts the Print Tool process. ] # prun 264
# ps | grep 264
264 console 0:03 printtool
#

Killing a Process (kill)

Sometimes it is necessary to stop (kill) a process. The process may be in an endless loop, or you may have started a large job that you want to stop before it is completed. You can kill any process that you own, and superuser can kill any processes in the system except for those with process IDs 0, 1, 2, 3, and 4.

Refer to kill(1) for more detailed information.

How to Kill a Process

1. (Optional) To kill a process belonging to another user, become superuser.

2. (Optional) Use output from the ps command to obtain the identification number of the process you want to display more information about.

   $ ps -e | grep process

   process

   Name of the process you want to display more information about.

   The process identification number is in the first column of the output.

3. Use the kill command to stop the process.

   $ kill [-9]PID ...

   -9	Ensures that the process terminates promptly.
   PID . . .	ID of the process or processes to stop.

4. Use the ps command to verify that the process has been stopped.

   $ ps -ef | grep PID ...

Managing Process Class Information

The listing below shows which classes are configured on your system, and the user priority range for the timesharing class. The possible classes are:

• System (SYS)

• Interactive (IA)

• Real-time (RT)

• Timesharing (TS)

  • The user-supplied priority ranges from -20 to +20.

  • The priority of a process is inherited from the parent process. This is referred to as the user-mode priority.

  • The system looks up the user-mode priority in the timesharing dispatch parameter table and adds in any nice or priocntl (user-supplied) priority and ensures a 0-59 range to create a global priority.

Changing the Scheduling Priority of Processes With priocntl

The scheduling priority of a process is the priority it is assigned by the process scheduler, according to scheduling policies. The dispadmin command lists the default scheduling policies. See "Scheduler Configuration", for information on the dispadmin command.

The priocntl command can be used to assign processes to a priority class and to manage process priorities. See the section called "How to Designate a Process Priority" for instructions on using the priocntl command to manage processes.

How to Display Basic Information About Process Classes

You can display process class and scheduling parameters with the priocntl -l command.

$ priocntl -l

Example--Getting Basic Information About Process Classes

The following example shows output from the priocntl -l command.

$ priocntl -l
CONFIGURED CLASSES
==================
 
SYS (System Class)
 
TS (Time Sharing)
	Configured TS User Priority Range: -20 through 20
 

How to Display the Global Priority of a Process

You can display the global priority of a process by using the ps command.

$ ps -ecl

The global priority is listed under the PRI column.

Example--Displaying the Global Priority of a Process

The following example shows output from ps -ecl. Data in the PRI column show that pageout has the highest priority, while sh has the lowest.

$ ps -ecl
 F S UID PID  PPID CLS PRI  ADDR      SZ  WCHAN    TTY      TIME   COMD
19 T 0   0    0    SYS 96   f00d05a8   0           ?        0:03  sched
 8 S 0   1    0    TS  50   ff0f4678 185  ff0f4848 ?       36:51   init
19 S 0   2    0    SYS 98   ff0f4018   0  f00c645c ?        0:01 pageout
19 S 0   3    0    SYS 60   ff0f5998   0  f00d0c68 ?      241:01 fsflush
 8 S 0   269  1    TS  58   ff0f5338 303  ff49837e ?        0:07    sac
 8 S 0   204  1    TS  43   ff2f6008  50  ff2f606e console  0:02     sh
 

How to Designate a Process Priority

1. Become superuser.

2. Start a process with a designated priority.

   # priocntl -e -c class-muserlimit-p pri command_name

   -e	Executes the command.
   -c class	Specifies the class within which to run the process. The default classes are TS (timesharing) or RT (real-time).
   -m userlimit	Specifies the maximum amount you can raise or lower your priority, when using the -p option.
   -p pri command_name	Lets you specify the relative priority in the RT class, for a real-time thread. For a timesharing process, the -p option lets you specify the user-supplied priority which ranges from -20 to +20.

3. Verify the process status by using the ps -ecl command.

   # ps -ecl | grep command_name

Example--Designating a Priority

The following example starts the find command with the highest possible user-supplied priority.

# priocntl -e -c TS -m 20 -p 20 find . -name core -print

# ps -ecl | grep find

How to Change Scheduling Parameters of a Timeshare Process

1. Become superuser.

2. Change the scheduling parameter of a running timeshare process.

   # priocntl -s -m userlimit [-p userpriority] -i idtype idlist

   -s	Lets you set the upper limit on the user priority range and change the current priority.
   -m userlimit	Specifies the maximum amount you can raise or lower your priority, when using the -p option.
   -p userpriority	Allows you to designate a priority.
   -i idtype idlist	Uses a combination of idtype and idlist to identify the process. The idtype specifies the type of ID, such as PID or UID.

3. Verify the process status by using the ps --ecl command.

   # ps -ecl | grep idlist

Example--Changing Scheduling Parameters of a Timeshare Process

The following example executes a command with a 500-millisecond time slice, a priority of 20 in the RT class, and a global priority of 120.

# priocntl -e -c RT -t 500 -p 20 myprog
# ps -ecl | grep myprog

How to Change the Class of a Process

1. (Optional) Become superuser.

   ---

   Note -

   You must be superuser or working in a real-time shell to change processes from, or to, real-time processes.

   ---

2. Change the class of a process.

   # priocntl -s -c class -i idtype idlist

   -s	Lets you set the upper limit on the user priority range and change the current priority.
   -c class	Specifies the class, TS or RT, to which you are changing the process.
   -i idtype idlist	Uses a combination of idtype and idlist to identify the process. The idtype specifies the type of ID, such as PID or UID.

3. Verify the process status by using the ps -ecl command.

   # ps -ecl | grep idlist

Example--Changing the Class of a Process

The following example changes all the processes belonging to user 15249 to real-time processes.

# priocntl -s -c RT -i uid 15249
# ps -ecl | grep 15249

If, as superuser, you change a user process to the real-time class, the user cannot subsequently change the real-time scheduling parameters (using priocntl -s).

Changing the Priority of a Timesharing Process With nice

The nice command is only supported for backward compatibility to previous Solaris releases. The priocntl command provides more flexibility in managing processes.

The priority of a process is determined by the policies of its scheduling class, and by its nice number. Each timesharing process has a global priority which is calculated by adding the user-supplied priority, which can be influenced by the nice or priocntl commands, and the system-calculated priority.

The execution priority number of a process is assigned by the operating system, and is determined by several factors, including its schedule class, how much CPU time it has used, and (in the case of a timesharing process) its nice number.

Each timesharing process starts with a default nice number, which it inherits from its parent process. The nice number is shown in the NI column of the ps report.

A user can lower the priority of a process by increasing its user-supplied priority. But only the superuser can lower a nice number to increase the priority of a process. This is to prevent users from increasing the priorities of their own processes, thereby monopolizing a greater share of the CPU.

Nice numbers range between 0 and +40, with 0 representing the highest priority. The default value is 20. Two versions of the command are available, the standard version, /usr/bin/nice, and a version that is part of the C shell.

How to Change the Priority of a Process

You can raise or lower the priority of a command or a process by changing the nice number. To lower the priority of a process:

The first and second commands increase the nice number by four units (the default); and the third command increases the nice by ten units, lowering the priority of the process.

The following commands raise the priority of the command by lowering the nice number.

To raise the priority of a process:

The above commands raise the priority of the command, command_name, by lowering the nice number. Note that in the second case, the two minus signs are required.

Process Troubleshooting

Here are some tips on obvious problems you may find:

• Look for several identical jobs owned by the same user. This may come as a result of running a script that starts a lot of background jobs without waiting for any of the jobs to finish.

• Look for a process that has accumulated a large amount of CPU time. You'll see this by looking at the TIME field. Possibly, the process is in an endless loop.

• Look for a process running with a priority that is too high. Type ps -c to see the CLS field, which displays the scheduler class of each process. A process executing as a real-time (RT) process can monopolize the CPU. Or look for a timeshare (TS) process with a high nice value. A user with superuser privileges may have bumped up the priorities of this process. The system administrator can lower the priority by using the nice command.

• Look for a runaway process--one that progressively uses more and more CPU time. You can see it happening by looking at the time when the process started (STIME) and by watching the cumulation of CPU time (TIME) for awhile.

Chapter 64 Monitoring Performance (Tasks)

This chapter describes procedures for monitoring system performance by using the vmstat, iostat, df, and sar commands. This is a list of the step-by-step instructions in this chapter.

• "How to Display Virtual Memory Statistics (vmstat)"

• "How to Display System Event Information"

• "How to Display Swapping Statistics"

• "How to Display Cache Flushing Statistics "

• "How to Display Interrupts Per Device"

• "How to Display Disk Utilization Information"

• "How to Display Extended Disk Statistics"

• "How to Display File System Information"

• "How to Check File Access (sar)"

• "How to Check Buffer Activity (sar)"

• "How to Check System Call Statistics (sar)"

• "How to Check Disk Activity (sar)"

• "How to Check Page-Out and Memory (sar)"

• "How to Check Kernel Memory Allocation (sar)"

• "How to Check Interprocess Communication (sar)"

• "How to Check Page-In Activity (sar)"

• "How to Check Queue Activity (sar)"

• "How to Check Unused Memory (sar)"

• "How to Check CPU Utilization (sar)"

• "How to Check System Table Status (sar)"

• "How to Check Swap Activity (sar)"

• "How to Check Terminal Activity (sar)"

• "How to Check Overall System Performance (sar)"

• "How to Set Up Automatic Data Collection "

Displaying Virtual Memory Statistics (vmstat)

You can use the vmstat command to report virtual memory statistics and such information about system events as CPU load, paging, number of context switches, device interrupts, and system calls. The vmstat command can also display statistics on swapping, cache flushing, and interrupts.

Refer to vmstat(1M) for a more detailed description of this command.

How to Display Virtual Memory Statistics (vmstat)

Collect virtual memory statistics using the vmstat command with a time interval.

$  vmstat n

Table 64-1 describes the fields in the vmstat output.

Example--Displaying Virtual Memory Statistics

The following example shows the vmstat display of statistics gathered at five-second intervals.

$ vmstat 5
 procs    memory            page             disk      faults     cpu
r b w  swap free re  mf  pi  po  fr de sr f0 s3 -- --  in  sy  cs us sy  id
0 0 8 28312  668  0   9   2   0   1  0  0  0  1  0  0  10  61  82  1  2  97
0 0 3 31940  248  0  10  20   0  26  0 27  0  4  0  0  53 189 191  6  6  88
0 0 3 32080  288  3  19  49   6  26  0 15  0  9  0  0  75 415 277  6 15  79
0 0 3 32080  256  0  26  20   6  21  0 12  1  6  0  0 163 110 138  1  3  96
0 1 3 32060  256  3  45  52  28  61  0 27  5 12  0  0 195 191 223  7 11  82
0 0 3 32056  260  0   1   0   0   0  0  0  0  0  0  0   4  52  84  0  1  99

How to Display System Event Information

Run vmstat -s to show the total of various system events that have taken place since the system was last booted.

$ vmstat -s
      0 swap ins
       0 swap outs
       0 pages swapped in
       0 pages swapped out 
 1329913 total address trans. faults taken
   25270 page ins
    3787 page outs
   38082 pages paged in
   13417 pages paged out
    3034 total reclaims
    3033 reclaims from free list
  335879 micro (hat) faults
  994034 minor (as) faults
   24210 major faults
  300634 copy-on-write faults
  141744 zero fill page faults
   34341 pages examined by the clock daemon
       5 revolutions of the clock hand
   28134 pages freed by the clock daemon
   11174 forks
    1259 vforks
    9086 execs 
11479519 cpu context switches 
95234544 device interrupts 
 1426943 traps 
 9100502 system calls
 1939346 total name lookups (cache hits 88%)
     496 toolong
  185566 user cpu
  977189 system cpu
92045953 idle cpu 
  130914 wait cpu

How to Display Swapping Statistics

Run vmstat -S to show swapping statistics.

$ vmstat -S
 procs   memory       page              disk       faults        cpu
 r b w swap free si so pi po fr de sr f0 s1 s3 -- in sy cs     us sy id
 0 0 0 6224  5536 0  0  0 0  0  0  0  0  0  0  0  2  9  12     0  1  99

The additional fields are described in Table 64-2.

The vmstat command truncates the output of both of these fields. Use the sar command to display a more accurate accounting of swap statistics.

How to Display Cache Flushing Statistics

Run vmstat -c to show cache flushing statistics for a virtual cache.

$ vmstat -c
flush statistics: (totals)
    usr      ctx     rgn    seg     pag     par
  14512    20201       0   1811 1857286  815505

It shows the total number of cache flushes since the last boot. The cache types are described in Table 64-3.

How to Display Interrupts Per Device

Run vmstat -i to show interrupts per device.

$ vmstat -i

Example--Displaying Interrupts Per Device

The following example shows output from the vmstat -i command.

$ vmstat -i
interrupt         total     rate
---------------------------------
clock         104638405      100
esp0            2895003        2
fdc0                  0        0
---------------------------------
Total         107533408      102

Displaying Disk Utilization Information

Use the iostat command to report statistics about disk input and output, and produces measures of throughput, utilization, queue lengths, transaction rates, and service time. For a detailed description of this command, refer to iostat(1M).

How to Display Disk Utilization Information

You can display disk activity information by using the iostat command with a time interval.

$ iostat 5
      tty     fd0     sd1     sd3     cpu
tin tout Kps tps serv Kps tps serv Kps tps serv us sy wt id
  0    0   0   0    0   1   0   79   0   0   58  0  1  0 99

The first line of output shows the statistics since the last boot. Each subsequent line shows the interval statistics. The default is to show statistics for the terminal (tty), disks (fd and sd), and CPU (cpu).

Table 64-4 describes the fields in the iostat command output.

Example--Displaying Disk Utilization Information

The following example shows disk statistics gathered every five seconds.

$ iostat 5
      tty          fd0           sd3       cpu
 tin tout bps tps serv  bps tps serv  us sy wt  id
   0    1   0   0    0    1   0 5640   0  1  0  98
   0   10   0   0    0    0   0    0   0  1  0  99
   0   10   0   0    0    0   0    0   0  1  0  99
   0   10   0   0    0   27   3  319   0  4  9  88
   0   10   0   0    0    2   0 5061   0  0  0  99
   0   10   0   0    0    0   0    0   0  0  0 100
   0   10   0   0    0    0   0    0   0  0  0 100
   0   10   0   0    0    0   0    0   0  0  0 100
   0   10   0   0    0    0   0    0   0  0  0 100
 

How to Display Extended Disk Statistics

Run iostat -xtc to get extended disk statistics.

$ iostat -xtc
               extended disk statistics               tty          cpu
disk  r/s w/s Kr/s Kw/s wait actv  svc_t  %w %b  tin tout us sy  wt id
sd0   0.2 1.7  1.0  9.7  0.0  0.1   39.8   0  3   0   9   1  6   9  85
sd1   0.5 2.5 10.6 21.0  0.0  0.1   26.6   0  5      
sd2   0.0 0.2  0.1  0.0  0.0  0.0  157.7   0  0

This command displays a line of output for each disk. The output fields are described in Table 64-5.

Displaying Disk Usage Statistics

Use the df command to show the amount of free disk space on each mounted disk. The usable disk space reported by df reflects only 90% of full capacity, as the reporting statistics leave a 10% head room above the total available space. This head room normally stays empty for better performance.

The percentage of disk space actually reported by df is used space divided by usable space.

If the file system is above 90% capacity, transfer files to a disk that is not as full by using cp, or to a tape by using tar or cpio; or remove the files.

For a detailed description of this command, refer to the df(1M) man page.

How to Display File System Information

Use the df -k command to display file system information in Kbytes.

$ df -k
Filesystem         kbytes   used   avail capacity  Mounted on
/dev/dsk/c0t3d0s0   192807   19980  153547    12%    /
 

Table 64-6 describes the df -k command output.

Example--Displaying File System Information

The following example shows output of the df -k command.

$ df -k
Filesystem        kbytes    used   avail capacity  Mounted on
/dev/dsk/c0t3d0s0   192807   19980  153547    12%    /
/dev/dsk/c0t3d0s6   384871  184385  162006    54%    /usr
/proc                    0       0       0     0%    /proc
fd                       0       0       0     0%    /dev/fd
/dev/dsk/c0t3d0s5   192807   54502  119025    32%    /opt
swap                184672     200  184472     1%    /tmp

Monitoring System Activities (sar)

Use the sar command to:

• Organize and view data about system activity

• Access system activity data on a special request basis

• Generate automatic reports to measure and monitor system performance, and special request reports to pinpoint specific performance problems. "Collecting System Activity Data Automatically (sar)" describes these tools.

For a detailed description of this command, refer to sar(1).

How to Check File Access (sar)

Display file access operation statistics with the sar -a command.

$ sar -a
SunOS venus 5.6 Generic sun4c 08/20/96
14:28:12    iget/s namei/s dirbk/s
14:29:12         0       2       1 
14:30:12         0       4       1 
14:31:12         0       3       1
Average          0       3       1 

The operating system routines reported are described in Table 64-7.

The larger the values reported, the more time the kernel is spending to access user files. The amount of time reflects how heavily programs and applications are using the file systems. The -a option is helpful for viewing how disk-dependent an application is.

How to Check Buffer Activity (sar)

Display buffer activity statistics with the sar -b command.

The buffer is used to cache metadata, which includes inodes, cylinder group blocks, and indirect blocks.

$ sar -b
SunOS venus 5.6 Generic sun4m    08/20/96
0:0:03 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
1:0:02     0       0     100       0       0      57       0   0

The buffer activities displayed by the -b option are described in Table 64-8. The most important entries are the cache hit ratios %rcache and %wcache, which measure the effectiveness of system buffering. If %rcache falls below 90, or if %wcache falls below 65, it may be possible to improve performance by increasing the buffer space.

Example--Checking Buffer Activity

The following example of sar -b output shows that the %rcache and %wcache buffers are not causing any slowdowns, because all the data is within acceptable limits.

$ sar -b
SunOS venus 5.6 Generic sun4m    08/20/96
14:28:12 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
14:29:12       0      14     100       6      17      67       0       0
14:30:12       0      12      99       6      16      65       0       0
14:31:12       0      12     100       6      16      65       0       0
 
Average        0      12     100       6      16      66       0       0

How to Check System Call Statistics (sar)

Display system call statistics by using the sar -c command.

$ sar -c
SunOS venus 5.6 Generic sun4m    08/20/96
00:00:03 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s
01:00:02       9       0       0   0.01   0.01      33       9

Table 64-9 describes the following system call categories reported by the -c option. Typically, reads and writes account for about half of the total system calls, although the percentage varies greatly with the activities that are being performed by the system.

Example--Checking System Call Statistics

The following example shows output from the sar -c command.

$ sar -c
Solaris mysys Solaris sun4c    08/22/96
14:28:12 scall/s sread/s swrit/s  fork/s  exec/s rchar/s wchar/s
14:29:12      17       2       2    0.28    0.28    2527    1542
14:30:12      25       2       1    0.50    0.47    1624     295
14:31:12      21       2       2    0.35    0.35    1812     703
 
Average       21       2       2    0.38    0.37    1987     847

How to Check Disk Activity (sar)

Display disk activity statistics with the sar -d command.

$ sar -d
SunOS venus 5.6 Generic sun4m    08/20/96
00:00:02 device %busy avque r+w/s blks/s avwait avserv
01:00:02 fd0        0   0.0     0      0    0.0    0.0
         sd1        0   0.0     0      0   19.6   35.4
         sd3        0   0.0     0      0   10.8   55.6

Table 64-10 describes the disk devices activities reported by the -d option. Note that queue lengths and wait times are measured when there is something in the queue. If %busy is small, large queues and service times probably represent the periodic efforts by the system to ensure that altered blocks are written to the disk in a timely fashion.

Examples--Checking Disk Activity

These two examples illustrate the sar -d output. The first example is from a computer with a non-SCSI (Small Computer System Interface, pronounced "scuzzy") integral disk; that is, a disk that does not use a SCSI interface. This example illustrates data being transferred from a hard disk (hdsk-0) to the floppy disk (fdsk-0).

$ sar -d
SunOS venus 5.6 Generic sun4m    08/20/96
13:46:28  device %busy avque r+w/s blks/s  avwait  avserv
13:46:58  hdsk-0    6   1.6     3      5    13.8    23.7
          fdsk-0   93   2.1     2      4   467.8   444.0
13:47:28  hdsk-0   13   1.3     4      8    10.8    32.3
          fdsk-0  100   3.1     2      5   857.4   404.1
13:47:58  hdsk-0   17    .7     2     41      .6    48.1
          fdsk-0  100   4.4     2      6  1451.9   406.5
Average   hdsk-0   12   1.2     3     18     8.4    34.7
          fdsk-0   98   3.2     2      5   925.7   418.2

The following example is from a computer with SCSI integral disks; that is, disks that use a SCSI interface. The example illustrates data being transferred from one SCSI hard disk (sd00-0) to another SCSI integral disk (sd00-1).

$ sar -d
SunOS venus 5.6 Generic sun4m    08/20/96
14:16:24  device %busy avque r+w/s blks/s  avwait  avserv
14:16:52 sd00-0      2   1.0     1      3     0.0    17.9
         sd00-1      6   1.1     3      5     2.0    23.9
14:17:21 sd00-0      2   1.0     1      2     0.0    19.6
         sd00-1      6   1.1     3      5     0.2    24.3
14:17:48 sd00-0      3   1.0     1      3     0.3    18.3
         sd00-1      7   1.1     3      5     1.3    25.4
14:18:15 sd00-0      3   1.0     1      3     0.0    17.2
         sd00-1      5   1.0     2      5     0.0    21.6
Average  sd00-0      2   1.0     1      3     0.1    18.2
         sd00-1      6   1.0     3      5     0.9    23.0

How to Check Page-Out and Memory (sar)

Use the sar -g option reports page-out and memory freeing activities (in averages).

$ sar -g
SunOS venus 5.6 Generic sun4m    08/20/96
00:00:03 pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf
01:00:02    0.00     0.00     0.00     0.00     0.00

The output displayed by sar -g is a good indicator of whether more memory may be needed. Use the ps -elf command to show the number of cycles used by the page daemon. A high number of cycles, combined with high values for pgfree/s and pgscan/s indicates a memory shortage.

sar -g also shows whether inodes are being recycled too quickly, causing a loss of reusable pages.

Output from the -g option is described in Table 64-11.

Example--Checking Page-Out and Memory

The following example shows output from the sar -g command.

$ sar -g
SunOS skisun 5.6 Generic sun4m    08/20/96
14:28:12  pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf
15:29:13     0.00     0.00     0.35     8.18   0.00
16:29:12     1.20     2.20     3.35     3.40   0.00

How to Check Kernel Memory Allocation (sar)

Use the sar -k command to report on the following activities of the Kernel Memory Allocator (KMA).

The KMA allows a kernel subsystem to allocate and free memory as needed. Rather than statically allocating the maximum amount of memory it is expected to require under peak load, the KMA divides requests for memory into three categories: small (less than 256 bytes), large (512 to 4 Kbytes), and oversized (greater than 4 Kbytes). It keeps two pools of memory to satisfy small and large requests. The oversized requests are satisfied by allocating memory from the system page allocator.

If you are investigating a system that is being used to write drivers or STREAMS that use KMA resources, then sar -k will likely prove useful. Otherwise, you will probably not need the information it provides. Any driver or module that uses KMA resources, but does not specifically return the resources before it exits, can create a memory leak. A memory leak causes the amount of memory allocated by KMA to increase over time. Thus, if the alloc fields of sar -k increase steadily over time, there may be a memory leak. Another indication of a memory leak is failed requests. If this occurs, a memory leak has probably caused KMA to be unable to reserve and allocate memory.

If it appears that a memory leak has occurred, you should check any drivers or STREAMS that may have requested memory from KMA and not returned it.

$ sar -k
SunOS venus 5.6 Generic sun4m    08/20/96
00:00:03  sml_mem   alloc  fail   lg_mem    alloc  fail  ovsz_alloc  fail
01:00:02  1245184  955332     0  3661824  2786336    0      2412544     0

Output from the -k option is described in Table 64-12.

Example--Checking Kernel Memory Allocation (sar)

The following is an example of sar -k output.

$ sar -k
SunOS skisun 5.6 Generic sun4m    08/20/96
 
14:28:12  sml_mem  alloc  fail  lg_mem  alloc  fail ovsz_alloc fail
14:29:12   95232   73472     0  311296  198656    0     180224    0
14:30:12   95232   75120     0  311296  198656    0     180224    0
14:31:12   95232   73600     0  311296  197632    0     180224    0
 
Average    95232   74064     0  311296  198314    0     180224    0
 

How to Check Interprocess Communication (sar)

Use the sar -m command to report interprocess communication activities.

$ sar -m 
SunOS venus 5.6 Generic sun4m    08/20/96
 00:00:03   msg/s  sema/s 
01:00:02    0.00    0.05 

These figures will usually be zero (0.00), unless you are running applications that use messages or semaphores.

The output from the -m option is described in Table 64-13.

Example--Checking Interprocess Communication

The following example shows output from the sar -m command.

$ sar -m
SunOS venus 5.6 Generic sun4m    08/20/96
14:28:12   msg/s  sema/s
14:29:12    0.00    0.00
14:30:12    0.00    0.00
14:31:12    0.00    0.00
 
Average     0.00    0.00

How to Check Page-In Activity (sar)

Use the sar -p command to report page-in activity which includes protection and translation faults.

$ sar -p
SunOS venus 5.6 Generic sun4m    08/20/96
00:00:03 atch/s pgin/s ppgin/s pflt/s vflt/s slock/s
01:00:02   0.00   0.00    0.00   0.49   1.20    0.00

The reported statistics from the -p option are described in Table 64-14.

Example--Checking Page-In Activity

The following example shows output from sar -p.

$ sar -p
SunOS venus 5.6 Generic sun4m    08/20/96
 
14:28:12  atch/s  pgin/s ppgin/s  pflt/s  vflt/s slock/s
14:29:12    1.17   12.87   12.87    5.67   11.28    1.15
14:30:12    1.67    7.08    7.08    9.12    6.33    0.67
14:31:12    1.37   12.48   12.48    6.83   10.78    1.03
 
Average     1.40   10.81   10.81    7.21    9.46    0.95
 

How to Check Queue Activity (sar)

Use the sar -q command to report the average queue length while the queue is occupied, and the percentage of time that the queue is occupied.

$ sar -q
SunOS venus 5.6 Generic sun4m    08/20/96
00:00:03 runq-sz %runocc swpq-sz %swpocc
01:00:02     1.1       0

The number of LWPs swapped out may greater than zero even if the system has an abundance of free memory. This happens when a sleeping LWP is swapped out and has not been awakened (for example, a process or LWP sleeping, waiting for the keyboard or mouse input).

Output from the -q option is described in Table 64-15.

Example--Checking Queue Activity

The following example shows output from the sar -q command. If %runocc is high (greater than 90 percent) and runq-sz is greater than 2, the CPU is heavily loaded and response is degraded. In this case, additional CPU capacity may be required to obtain acceptable system response.

$ sar -q
SunOS venus 5.6 Generic sun4m    08/20/96
14:28:12 runq-sz %runocc swpq-sz %swpocc
14:29:12     1.2      53    1       100
14:30:12     1.3      38
14:31:12     1.1      37
 
Average      1.2      43

How to Check Unused Memory (sar)

Use the sar -r command to report the number of memory pages and swap-file disk blocks that are currently unused.

$ sar -r
SunOS venus 5.6 Generic sun4m    08/20/96
00:00:03 freemem freeswap
01:00:02     983   187590

Output from the -r option is described in Table 64-16.

Example--Checking Unused Memory

The following example shows output from the sar -r command.

$ sar -r
SunOS venus 5.6 Generic sun4m    08/20/96
14:28:12 freemem freeswap
14:29:12     268    3034
14:30:12     351    3009
14:31:12     297    3033
 
Average      306    3025
 

How to Check CPU Utilization (sar)

Display CPU utilization with the sar -u command.

$ sar -u
SunOS venus 5.6 Generic sun4m    08/20/96
00:00:03    %usr    %sys    %wio %idle
01:00:02       0       1       0    99

(The sar command without any options is equivalent to sar -u.) At any given moment, the processor is either busy or idle. When busy, the processor is in either user or system mode. When idle, the processor is either waiting for I/O completion or "sitting still" with no work to do.

Output from the -u option is described in Table 64-17.

A high %wio generally means a disk slowdown has occurred.

Example--Checking CPU Utilization

The following example shows output from the sar -u command.

$ sar -u
SunOS venus 5.6 Generic sun4m    08/20/96
14:28:12    %usr    %sys    %wio   %idle
14:29:12      22      27      18      32
14:30:12       6      24      13      57
14:31:12       8      28      19      45
 
Average       12      27      17      45
 

How to Check System Table Status (sar)

Use the sar -v command to report the status of the process table, inode table, file table, and shared memory record table.

$ sar -v
SunOS venus 5.6 Generic sun4m    08/20/96
00:00:03 proc-sz      ov   inod-sz   ov  file-sz ov  lock-sz
01:00:02  56/426       0  1311/1311   0  372/372  0   0/0 

Output from the -v option is described in Table 64-18.

Example--Checking System Table Status

The following example shows output from the sar -v command. This example shows that all tables are large enough to have no overflows. These tables are all dynamically allocated based on the amount of physical memory.

$ sar -v
SunOS venus 5.6 Generic sun4m    08/20/96
14:28:12 proc-sz ov inod-sz ov file-sz ov lock-sz
14:29:12  28/200  0 297/300  0  63/0    0  6/0
14:30:12  30/200  0 297/300  0  65/0    0  6/0
14:31:12  28/200  0 296/300  0  63/0    0  6/0

How to Check Swap Activity (sar)

Use the sar -w command to report swapping and switching activity.

$ sar -w
SunOS venus 5.6 Generic sun4m    08/20/96
00:00:03  swpin/s  bswin/s  swpot/s  bswot/s  pswch/s
01:00:02     0.00      0.0     0.00      0.0       12

Target values and observations are described in Table 64-19.

All process swap-ins include process initialization.

Example--Checking Swap Activity

The following example shows output from the sar -w command.

$ sar -w
SunOS venus 5.6 Generic sun4m    08/20/96
14:28:12 swpin/s pswin/s swpot/s pswot/s pswch/s
14:29:12    0.00     0.0    0.00     0.0      22
14:30:12    0.00     0.0    0.00     0.0      12
14:31:12    0.00     0.0    0.00     0.0      18
 
Average     0.00     0.0    0.00     0.0      18

How to Check Terminal Activity (sar)

Use the sar -y command to monitor terminal device activities.

$ sar -y
SunOS saturn 5.6 prefcs3 sun4c   08/20/96
00:00:03 rawch/s canch/s outch/s rcvin/s xmtin/s mdmin/s
01:00:02       0       0       0       0       0       0

If you have a lot of terminal I/O, you can use this report to determine if there are any bad lines. The activities recorded are defined in Table 64-20.

The number of modem interrupts per second (mdmin/s) should be close to zero, and the receive and transmit interrupts per second (xmtin/s and rcvin/s) should be less than or equal to the number of incoming or outgoing characters, respectively. If this is not the case, check for bad lines.

Example--Checking Terminal Activity

The following example shows output from the sar -y command.

$ sar -y
SunOS venus 5.6 Generic sun4m    08/20/96
14:28:12 rawch/s canch/s outch/s rcvin/s xmtin/s mdmin/s
14:29:12       0       1     157       1       3       0
14:30:12       0       2      34       2       2       0
14:31:12       0       1      11       1       2       0
 
Average        0       1      67       1       2       0

How to Check Overall System Performance (sar)

Use the sar -A command to display a view of overall system performance.

This provides a more global perspective. If data from more than one time segment is shown, the report includes averages.

Collecting System Activity Data Automatically (sar)

Three commands are involved in automatic system activity data collection: sadc, sa1, and sa2.

The sadc data collection utility periodically collects data on system activity and saves it in a file in binary format--one file for each 24-hour period. You can set up sadc to run periodically (usually once each hour), and whenever the system boots to multiuser mode. The data files are placed in the directory /usr/adm/sa. Each file is named sadd, where dd is the current date. The format of the command is as follows:

/usr/lib/sa/sadc [t n] [ofile]

The command samples n times with an interval of t seconds (t should be greater than 5 seconds) between samples. It then writes, in binary format, to the file ofile, or to standard output. If t and n are omitted, a special file is written once.

Running sadc When Booting

The sadc command should be run at system boot time in order to record the statistics from when the counters are reset to zero. To make sure that sadc is run at boot time, the /etc/init.d/perf file must contain a command line that writes a record to the daily data file.

The command entry has the following format:

su sys -c "/usr/lib/sa/sadc /usr/adm/sa/sa`date +5d`"
 

Running sadc Periodically With sa1

To generate periodic records, you need to run sadc regularly. The simplest way to do this is by putting a line into the /var/spool/cron/sys file, which calls the shell script, sa1. This script invokes sadc and writes to the daily data files, /var/adm/sa/sadd. It has the following format:

/usr/lib/sa/sa1 [t n]

The arguments t and n cause records to be written n times at an interval of t seconds. If these arguments are omitted, the records are written only one time.

Producing Reports With sa2

Another shell script, sa2, produces reports rather than binary data files. The sa2 command invokes the sar command and writes the ASCII output to a report file.

Collecting System Activity Data (sar)

The sar command can be used either to gather system activity data itself or to report what has been collected in the daily activity files created by sadc.

The sar command has the following formats:

sar [-aAbcdgkmpqruvwy] [-o file] t [n] 

sar [-aAbcdgkmpqruvwy] [-s time] [-e time] [-i sec] [-f file]

The sar command below samples cumulative activity counters in the operating system every t seconds, n times. (t should be 5 seconds or greater; otherwise, the command itself may affect the sample.) You must specify a time interval between which to take the samples; otherwise, the command operates according to the second format. The default value of n is 1. The following example takes two samples separated by 10 seconds. If the -o option is specified, samples are saved in file in binary format.

$ sar -u 10 2

Other important information about the sar command:

• With no sampling interval or number of samples specified, sar extracts data from a previously recorded file, either the one specified by the -f option or, by default, the standard daily activity file, /var/adm/sa/sadd, for the most recent day.

• The -s and -e options define the starting and ending times for the report. Starting and ending times are of the form hh[:mm[:ss]] (where h, m, and s represent hours, minutes, and seconds).

• The -i option specifies, in seconds, the intervals between record selection. If the -i option is not included, all intervals found in the daily activity file are reported.

Table 64-21 lists the sar options and their actions.

If no option is used, it is equivalent to calling the command with the -u option.

How to Set Up Automatic Data Collection

1. Become superuser.

2. Using the editor of your choice, open the /etc/init.d/perf file, which contains the sadc start-up instructions. Verify that the following lines are uncommented:

   MATCH=`who -r|grep -c "[234][ ]*0[ ]*[S1]"` if [ ${MATCH} -eq 1 ] then su sys -c "/usr/lib/sa/sadc /var/adm/sa/sa`date +%d`" fi

   This version of the sadc command writes a special record that marks the time when the counters are reset to zero (boot time). The sadc output is put into the file sadd (where dd is the current date), which acts as the daily system activity record.

3. Using the editor of your choice, open the /var/spool/cron/crontabs/sys file (the system crontab file). Uncomment the following lines:

   # 0 * * * 0-6 /usr/lib/sa/sa1 # 20,40 8-17 * * 1-5 /usr/lib/sa/sa1

   The first entry writes a record to /var/adm/sa/sadd on the hour, every hour, seven days a week.

   The second entry writes a record to /var/adm/sa/sadd twice each hour during peak working hours: at 20 minutes and 40 minutes past the hour, from 8 a.m. to 5 p.m., Monday through Friday.

   Thus, these two crontab entries cause a record to be written to /var/adm/sa/sadd every 20 minutes from 8 a.m. to 5 p.m., Monday through Friday, and every hour on the hour otherwise. You can change these defaults to meet your needs.

Chapter 65 Monitoring Network Performance (Tasks)

This chapter describes the how to monitor network performance. This is a list of the step-by-step instructions in this chapter.

• "How to Check the Response of Hosts on the Network"

• "How to Send Packets to Hosts on the Network"

• "How to Capture Packets From the Network"

• "How to Check the Network Status"

• "How to Display NFS Server and Client Statistics"

Monitoring Network Performance

Table 65-1 describes the commands available for monitoring network performance.

How to Check the Response of Hosts on the Network

Check the response of hosts on the network with the ping command.

$ ping hostname

If you suspect a physical problem, you can use ping to find the response time of several hosts on the network. If the response from one host is not what you would expect, you can investigate that host. Physical problems could be caused by:

• Loose cables or connectors

• Improper grounding

• Missing termination

• Signal reflection

For more information about this command, see ping(1M).

Examples--Checking the Response of Hosts on the Network

The simplest version of ping sends a single packet to a host on the network. If it receives the correct response, it prints the message host is alive.

$ ping elvis
elvis is alive

With the -s option, ping sends one datagram per second to a host. It then prints each response and the time it took for the round trip. For example:

$ ping -s pluto
64 bytes from pluto (123.456.78.90): icmp_seq=0. time=10. ms
64 bytes from pluto (123.456.78.90): icmp_seq=5. time=0. ms
64 bytes from pluto (123.456.78.90): icmp_seq=6. time=0. ms
^C
----pluto PING Statistics----
8 packets transmitted, 8 packets received, 0% packet loss
 
round-trip (ms) min/avg/max = 0/2/10

How to Send Packets to Hosts on the Network

Test the reliability of your packet sizes with the spray command.

$ spray [ -c count -d interval -l packet_size] hostname

-c count

-d interval

-l packet_size

For more information about this command, see spray(1M).

Example--Sending Packets to Hosts on the Network

The following example sends 100 packets to a host (-c 100) with each packet having a size of 2048 bytes (-l 2048). The packets are sent with a delay time of 20 microseconds between each burst (-d 20).

$ spray -c 100 -d 20 -l 2048 pluto
sending 100 packets of length 2048 to pluto ...
no packets dropped by pluto
279 packets/sec, 573043 bytes/sec

How to Capture Packets From the Network

To capture packets from the network and trace the calls from each client to each server, use snoop. This command provides accurate time stamps that allow some network performance problems to be isolated quickly. For more information, see snoop(1M).

# snoop

Dropped packets could be caused by insufficient buffer space, or an overloaded CPU.

How to Check the Network Status

Display network status information, such as statistics about the state of network interfaces, routing tables, and various protocols, with the netstat command.

$ netstat [-i] [-r] [-s]

For more information, see netstat(1M).

Examples--Checking the Network Status

The following example shows output from the netstat -i command, which displays the state of the interfaces used for TCP/IP traffic.

$ netstat -i
Name  Mtu  Net/Dest    Address      Ipkts  Ierrs Opkts  Oerrs Collis Queue
lo0   8232 software    localhost     1280   0     1280     0       0    0
le0   1500 loopback    venus      1628480   0   347070    16   39354    0

This display shows how many packets a machine has transmitted and received on each interface. A machine with active network traffic should show both Ipkts and Opkts continually increasing.

Calculate the network collisions rate by dividing the number of collision counts (Collis) by the number of out packets (Opkts). In the above example, the collision rate is 3.5 percent. A network-wide collision rate greater than 5 to 10 percent can indicate a problem.

Calculate the input packet error rate by dividing the number of input errors by the total number of input packets (Ierrs/Ipkts). The output packet error rate is the number of output errors divided by the total number of output packets (Oerrs/Opkts). If the input error rate is high (over 0.25 percent), the host may be dropping packets.

The following example shows output from the netstat -s command, which displays the per-protocol statistics for the UDP, TCP, ICMP, and IGMP protocols.

UDP
	udpInDatagrams      =196543	udpInErrors         =     0
	udpOutDatagrams     =187820
 
TCP	
 tcpRtoAlgorithm     =     4	tcpRtoMin           =   200
	tcpRtoMax           = 60000	tcpMaxConn          =    -1
	tcpActiveOpens      = 26952	tcpPassiveOpens     =   420
	tcpAttemptFails     =  1133	tcpEstabResets      =     9
	tcpCurrEstab        =    31	tcpOutSegs          =3957636
	tcpOutDataSegs      =2731494	tcpOutDataBytes     =1865269594
	tcpRetransSegs      = 36186	tcpRetransBytes     =3762520
	tcpOutAck           =1225849	tcpOutAckDelayed    =165044
	tcpOutUrg           =     7	tcpOutWinUpdate     =   315
	tcpOutWinProbe      =     0	tcpOutControl       = 56588
	tcpOutRsts          =   803	tcpOutFastRetrans   =   741
	tcpInSegs           =4587678
	tcpInAckSegs        =2087448	tcpInAckBytes       =1865292802
	tcpInDupAck         =109461	tcpInAckUnsent      =     0
	tcpInInorderSegs    =3877639	tcpInInorderBytes   =-598404107
	tcpInUnorderSegs    = 14756	tcpInUnorderBytes   =17985602
	tcpInDupSegs        =    34	tcpInDupBytes       = 32759
	tcpInPartDupSegs    =   212	tcpInPartDupBytes   =134800
	tcpInPastWinSegs    =     0	tcpInPastWinBytes   =     0
	tcpInWinProbe       =   456	tcpInWinUpdate      =     0
	tcpInClosed         =    99	tcpRttNoUpdate      =  6862
	tcpRttUpdate        =435097	tcpTimRetrans       = 15065
	tcpTimRetransDrop   =    67	tcpTimKeepalive     =   763
	tcpTimKeepaliveProbe=     1	tcpTimKeepaliveDrop =     0
 
IP	
 ipForwarding        =     2	ipDefaultTTL        =   255
	ipInReceives        =11757234	ipInHdrErrors       =     0
	ipInAddrErrors      =     0	ipInCksumErrs       =     0
	ipForwDatagrams     =     0	ipForwProhibits     =     0
	ipInUnknownProtos   =     0	ipInDiscards        =     0
	ipInDelivers        =4784901	ipOutRequests       =4195180
	ipOutDiscards       =     0	ipOutNoRoutes       =     0
	ipReasmTimeout      =    60	ipReasmReqds        =  8723
	ipReasmOKs          =  7565	ipReasmFails        =  1158
	ipReasmDuplicates   =     7	ipReasmPartDups     =     0
	ipFragOKs           = 19938	ipFragFails         =     0
	ipFragCreates       =116953	ipRoutingDiscards   =     0
	tcpInErrs           =     0	udpNoPorts          =6426577
	udpInCksumErrs      =     0	udpInOverflows      =   473
	rawipInOverflows    =     0
 
ICMP	
 icmpInMsgs          =490338	icmpInErrors        =     0
	icmpInCksumErrs     =     0	icmpInUnknowns      =     0
	icmpInDestUnreachs  =   618	icmpInTimeExcds     =   314
	icmpInParmProbs     =     0	icmpInSrcQuenchs    =     0
	icmpInRedirects     =   313	icmpInBadRedirects  =     5
	icmpInEchos         =   477	icmpInEchoReps      =    20
	icmpInTimestamps    =     0	icmpInTimestampReps =     0
	icmpInAddrMasks     =     0	icmpInAddrMaskReps  =     0
	icmpInFragNeeded    =     0	icmpOutMsgs         =   827
	icmpOutDrops        =   103	icmpOutErrors       =     0
	icmpOutDestUnreachs =    94	icmpOutTimeExcds    =   256
	icmpOutParmProbs    =     0	icmpOutSrcQuenchs   =     0
	icmpOutRedirects    =     0	icmpOutEchos        =     0
	icmpOutEchoReps     =   477	icmpOutTimestamps   =     0
	icmpOutTimestampReps=     0	icmpOutAddrMasks    =     0
	icmpOutAddrMaskReps =     0	icmpOutFragNeeded   =     0
	icmpInOverflows     =     0
IGMP:
        0 messages received
        0 messages received with too few bytes
        0 messages received with bad checksum
        0 membership queries received
        0 membership queries received with invalid field(s)
        0 membership reports received
        0 membership reports received with invalid field(s)
        0 membership reports received for groups to which we belong
        0 membership reports sent

The following example shows output from the netstat -r command, which displays the IP routing table.

Routing Table:
  Destination        Gateway           Flags  Ref   Use    Interface
------------------ -------------------- ----- ----- ------ ---------
localhost            localhost             UH       0   2817  lo0
earth-bb             pluto                 U        3  14293  le0
224.0.0.0            pluto                 U        3      0  le0
default              mars-gate             UG       0  14142  
 

The fields in the netstat -r report are described in Table 65-2.

How to Display NFS Server and Client Statistics

The NFS distributed file service uses a remote procedure call (RPC) facility which translates local commands into requests for the remote host. The remote procedure calls are synchronous. That is, the client application is blocked or suspended until the server has completed the call and returned the results. One of the major factors affecting NFS performance is the retransmission rate.

If the file server cannot respond to a client's request, the client retransmits the request a specified number of times before it quits. Each retransmission imposes system overhead, and increases network traffic. Excessive retransmissions can cause network performance problems. If the retransmission rate is high, you could look for:

• Overloaded servers that take too long to complete requests

• An Ethernet interface dropping packets

• Network congestion which slows the packet transmission

Table 65-3 describes the nfsstat options to display client and server statistics.

Use nfsstat -c to show client statistics, and nfsstat -s to show server statistics. Use netstat -m to display network statistics for each file system. For more information, see nfsstat(1M).

Examples--Displaying NFS Server and Client Statistics

The following example displays RPC and NFS data for the client, pluto.

$ nfsstat -c
 
        Client rpc:
Connection oriented:
calls    badcalls  badxids  timeouts newcreds  badverfs   timers     
1595799  1511      59       297      0         0          0          
cantconn nomem     interrupts 
1198      0         7          
Connectionless:
calls    badcalls  retrans  badxids  timeouts  newcreds   badverfs   
80785    3135      25029    193      9543      0          0          
timers   nomem     cantsend   
17399    0         0          
 
Client nfs:
calls    badcalls  clgets   cltoomany  
1640097  3112      1640097  0          
Version 2: (46366 calls)
null     getattr   setattr  root     lookup     readlink  read       
0 0%     6589 14%  2202 4%  0 0%     11506 24%  0 0%      7654 16%   
wrcache  write     create   remove   rename     link      symlink    
0 0%     13297 28% 1081 2%  0 0%     0 0%       0 0%      0 0%       
mkdir    rmdir     readdir  statfs     
24 0%    0 0%      906 1%   3107 6%    
Version 3: (1585571 calls)
null    getattr    setattr  lookup     access     readlink  read     
0 0%    508406 32% 10209 0% 263441 16% 400845 25% 3065 0%  117959 7%
write    create     mkdir    symlink    mknod    remove   rmdir 
69201 4% 7615 0%    42 0%    16 0%      0 0%     7875 0%  51 0%      
rename   link       readdir  readdir+   fsstat   fsinfo   pathconf   
929 0%   597 0%     3986 0%  185145 11% 942 0%   300 0%   583 0%     
commit     
4364 0%    
 
Client nfs_acl:
Version 2: (3105 calls)
null       getacl     setacl     getattr    access     
0 0%       0 0%       0 0%       3105 100%  0 0%       
Version 3: (5055 calls)
null       getacl     setacl     
0 0%       5055 100%  0 0%    

The output of the nfsstat -c command is described in Table 65-4.

The following example shows output from the nfsstat -m command.

pluto$ nfsstat -m
/usr/man from pluto:/export/svr4/man
Flags: vers=2,proto=udp,auth=unix,hard,intr,dynamic,
        rsize=8192, wsize=8192,retrans=5
 Lookups: srtt=13 (32ms), dev=10 (50ms), cur=6 (120ms)
 All:     srtt=13 (32ms), dev=10 (50ms), cur=6 (120ms)

This output of the nfsstat -m command, which is displayed in milliseconds, is described in Table 65-5.

If you suspect that the hardware components of your network are creating problems, you need to look carefully at the cabling and connectors.

Chapter 66 Tuning Kernel Parameters (Tasks)

This chapter describes the procedures for tuning kernel parameters. This is a list of the step-by-step instructions in this chapter.

• "Listing the Kernel Parameters"

• "How to Change the Value of a Kernel Parameter"

• "How to Set the Value of a Kernel Module Variable"

• "How to Tune the Interprocess Communication Parameters"

• "How to Tune Memory Management Parameters"

• "How to Tune Miscellaneous Parameters"

Listing the Kernel Parameters

Display the current kernel parameters values by using the sysdef -i command.

# sysdef -i
* Hostid
  53001b80
*
* sun4m Configuration
* Devices
	packages (driver not attached)
		disk-label (driver not attached)
	   deblocker (driver not attached)
  	obp-tftp (driver not attached)
              .
              .
              .
options, instance #0
aliases (driver not attached)
openprom (driver not attached)
iommu, instance #0
	sbus, instance #0
		espdma, instance #0
			esp, instance #0
				sd (driver not attached)
				st (driver not attached)
				      .
               .
               .

How to Change the Value of a Kernel Parameter

1. Become superuser.

2. Add a line to the /etc/system file in the form:

   set parameter=value

3. Verify the kernel parameter change.

   # grep parameter /etc/system

4. Reboot the system.

   The kernel parses the /etc/system file during autoconfiguration and overrides the default value for the parameters specified in this file.

Example--Changing the Value of a Kernel Parameter

The following line in the /etc/system file sets the value of the max_nprocs to 500 parameter.

set max_nprocs=500

How to Set the Value of a Kernel Module Variable

1. Become superuser.

2. Add a line to the /etc/system file in the form:

   set module_name:variable=value

3. Verify the kernel module variable change.

   # grep module_name /etc/system

4. Reboot the system.

   The kernel parses the /etc/system file during autoconfiguration and overrides the default value for the parameters specified in this file.

Example--Setting the Value of a Kernel Module Variable

The following line in the /etc/system file sets the value of the msginfo_msgmap parameter in the msgsys module to 150.

set msgsys:msginfo_msgmap=150  

Buffer Cache Parameters

The bufhwm parameter specifies the maximum size for buffer cache memory usage expressed in units of 1,000 bytes. The default is 2% of physical memory. Use sar(1M) to measure the buffer cache statistics.

UFS Parameters

Table 66-1 describes the tunable UFS parameters.

STREAMS Parameters

Table 66-2 describes the tunable STREAMS parameters.

Interprocess Communication (IPC) Parameters

Table 66-3 describes the tunable interprocess communication parameters.

How to Tune the Interprocess Communication Parameters

1. Become superuser.

2. Add a line to the /etc/system file using the syntax described in Table 66-4.

   Table 66-4 Tuning Interprocess Communication Parameters

   Parameter Type	Parameter	Tuning Syntax
   Message Queue	msgsys	set msgsys:msginfo_variable = value
   Semaphore Facility	semsys	set semsys:seminfo_variable=value
   Shared Memory	shmsys	set shmsys:shminfo_variable=value

3. Verify the kernel parameter change.

   # grep parameter /etc/system

4. Reboot the system.

   The kernel parses the /etc/system file during autoconfiguration and overrides the default value for the parameters specified in this file.

Memory Management Parameters

Table 66-5 describes the tunable memory management parameters.

Since the Solaris 2.4 release, the tune_t_gpgslo parameter has been replaced by a more complicated criteria for swapping based on the number of runnable threads.

The freemem parameter is defined in pages. Utilities like vmstat translates freemem into bytes from pages.

How to Tune Memory Management Parameters

1. Become superuser.

2. Add a line to the /etc/system file using the following syntax.

   set tune:variable=value

3. Verify the kernel parameter change.

   # grep parameter /etc/system

4. Reboot the system.

   The kernel parses the /etc/system file during autoconfiguration and overrides the default value for the parameters specified in this file.

Miscellaneous Parameters

Table 66-6 describes tunable miscellaneous parameters.

How to Tune Miscellaneous Parameters

1. Become superuser.

2. Add a line to the /etc/system file using the following syntax.

   set parameter=value

3. Verify the kernel parameter change.

   # grep parameter /etc/system

4. Reboot the system.

   If you changed device related kernel parameters, you need to use the -r option when booting the system. When the system boots, the kernel parses the /etc/system file during autoconfiguration and overrides the default value for the parameters specified in this file.

Example--Tuning Miscellaneous Parameters

The following line in the /etc/system file sets the value of the pt_cnt parameter to 200.

set pt_cnt=200 

Chapter 67 The Scheduler (Reference)

This chapter contains reference information for the SunOS 5.x scheduler. This is a list of the overview information in this chapter.

• "About the Scheduler"

• "Scheduler Class Policies"

• "Scheduler Configuration"

About the Scheduler

The scheduler (or dispatcher) is the portion of the kernel that controls the allocation of the CPU to processes. It determines when processes run and for how long, depending on their assigned priorities. Priorities are based on scheduling class and process behavior. Four scheduling classes are supported by default: timesharing, system, real-time and interactive.

The scheduler has an overriding effect on the performance of a system.

The fundamental scheduling entity is the kernel thread. For single-threaded processes, scheduling the kernel thread is synonymous with process scheduling.

The SunOS 5.x scheduler controls the order in which processes run and the amount of CPU time each process may use before another process can run.

The scheduler allocates CPU time to processes according to the scheduling policies defined for each scheduling class. Associated with each scheduling class is a set of priority levels or queues. Ready-to-run processes are moved among these queues. Within a class, you can view these queues as a contiguous set of priority levels. These priority levels are mapped into a set of global scheduling priorities.

The global priority of a process determines when it runs--the scheduler runs the process with the highest global priority that is ready to run. Processes with numerically higher priorities run first, and processes with the same priority run using a round robin scheduling policy.

Once the scheduler assigns a process to a CPU, the process runs until one of the following events occur:

• The process uses up its time slice.

• The process blocks waiting for an event (for example, I/O) or a suspended lock.

• The process is preempted by a higher-priority process.

By default, all real-time processes have higher priorities than any system process, and all system processes have higher priorities than any timesharing process.

A process inherits its scheduler parameters from its parent process, including its scheduler class and its priority within that class. A process changes class only from a user request (with the priocntl command or system call). The system manages the priority of a process based on user requests and the policy associated with the scheduling class of the process.

Scheduler Class Policies

The following sections describe the scheduling policies of the three default classes: timesharing, system, and real-time.

Timesharing Class Policies

In the default configuration, the initialization process (init) belongs to the timesharing class. Because processes inherit their scheduler parameters, all user login shells--and consequently the processes run from those shells--begin as timesharing processes.

The goal of the timesharing policy is to provide good response time for interactive processes and good throughput for processes that use a lot of CPU time. The scheduler tries to divide the CPU's time fairly between processes, subject to the priorities associated with the processes. Those with higher priorities get more attention than those with lower priorities. However, to prevent any one job (process) from hogging the CPU, the scheduler can move jobs from high priorities to low priorities and vice versa.

The scheduler switches CPU allocation frequently enough to provide good response time, but not so frequently that it spends too much time doing the switching. Time slices are typically on the order of a few hundredths of a second.

The timesharing policy changes priorities dynamically and assigns time slices of different lengths. Once a process has started, its timesharing priority varies according to how much CPU time it's getting, how much time it's spending in queues, and other factors. The scheduler raises the priority of a process that "sleeps." (A process sleeps, for example, when it starts an I/O operation such as a terminal read or a disk read.) Entering sleep states frequently is characteristic of interactive tasks such as editing and running simple shell commands. On the other hand, the timesharing policy lowers the priority of a process that uses the CPU for long periods without sleeping.

The default timesharing policy gives larger time slices to processes with lower priorities. A process with a low priority is likely to be stuck in the CPU. Other processes get the CPU first, but when a lower-priority process finally gets the CPU, it gets a bigger chunk of time. If a higher-priority process becomes ready to run during a time slice, however, it preempts the running process.

The scheduler manages timesharing processes using parameters in the timesharing parameter table ts_dptbl. This table contains information specific to the timesharing class. It is automatically loaded into core memory from the TS_DPTBL loadable module located in the /kernel/sched directory.

System Class Policies

The system class uses a fixed-priority policy to run kernel processes such as servers, and housekeeping processes such as the page daemon. Their priorities are not dynamically adjusted like timesharing processes. The system class is reserved for use by the kernel, and users may neither add nor remove a process from the system class. Priorities for system-class processes are set up in the kernel code for the kernel processes, and, once established, these priorities do not change. (User processes running in kernel mode are not in the system class.)

Real-Time Class Policies

The SunOS 5.x operating system uses a real-time scheduling policy as well as a timesharing policy. Real-time scheduling allows users to set fixed priorities on a per-process basis, so that critical processes can run in predetermined order. The real-time scheduler never moves jobs between priorities. Real-time priorities change only when a user requests a change (using the priocntl command). Contrast this fixed-priority policy with the timesharing policy, in which the system changes priorities to provide good interactive response time.

The user process with the highest real-time priority always gets the CPU as soon as it can be run, even if other processes are ready to run. An application can be written so that its real-time processes have a guaranteed response time from the operating system.

As long as there is a real-time process ready to run, no process and no timesharing process runs. Other real-time processes can run only if they have a higher priority. Real-time processes managed carelessly can have a dramatic negative effect on the performance of timesharing processes.

The real-time policy gives higher-priority processes smaller time slices, by default. The higher priorities are allocated to real-time processes that are driven by external events. The operating system must be able to respond instantly to I/O. The lower-priority real-time processes are those that need more computation time. If a process with the highest priority uses up its time slice, it runs again because there is no process with a higher priority to pre-empt it.

The scheduler manages real-time processes by using parameters in the real-time parameter table rt_dptbl. This table contains information specific to the real-time class. It is automatically loaded into core from the RT_DPTBL loadable module located in the /kernel/sched directory.

Scheduler Configuration

This section describes the parameters and tables that control the scheduler configuration. A basic assumption is that your work load is reasonable for your system resources, such as CPU, memory, and I/O. If your resources are inadequate to meet the demands, reconfiguring the scheduler won't help.

You can display or change (fine tune) the scheduler parameters in a running system for both the timesharing and real-time classes by using the dispadmin command. Changes made by the dispadmin command do not survive a reboot. To make permanent changes in scheduler configuration, you must change the scheduler parameter tables in the appropriate loadable module: TS_DPTBL or RT_DPTBL provided in the /kernel/sched directory. See ts_dptbl(4) and rt_dptbl(4) for instructions on replacing these modules.

The primary user command for controlling process scheduling is priocntl(1). With this command, a user can start a process at a specified priority or manipulate the priorities of running processes. You can find out what classes are configured on your system with the priocntl -l command. The primary function call for controlling process scheduling is priocntl(2).

See Chapter 63, Managing Processes (Tasks), for examples of using the priocntl command. See System Interface Guide for a detailed descriptions of real-time programming, and the dispadmin(1M) and priocntl(1) commands.

Default Global Priorities

The following table shows the scheduling order and ranges of global priorities for each scheduler class.

How Global Priorities Are Constructed

When your operating system is built, it constructs the global priorities from the tunable parameters and scheduler parameter tables described in the following sections. There isn't any command that will show you this complete global priority table. However, the dispadmin command displays the priorities (from 0 to n) specific to the real-time and timesharing classes. You can display the global priority of an active process with the ps -cl command.

Initial Global Priorities of Processes

A timesharing process inherits its scheduling class and priority from its parent process. The init process is the first process to entire the timesharing class.

System processes initially run with a priority that depends on the process's importance (which is programmed into the kernel). The most important system processes start with a priority at or near the top of the system class range.

Tunable Parameters

This section describes the tunable parameters that control scheduler configuration. To change any of these kernel parameters, enter a line in the /etc/system file with the format:

set parameter=value

See system(4) for more information.

The parameters described in this section control aspects of process scheduling, timesharing policy, and real-time policy.

The initial priority of a real-time process is determined when the process is put into the real-time scheduling class.

The priocntl -p command is used to specify the relative priority within the real-time class.

This is added to the base priority of the real-time class, which by default is 100. For example:

priocntl -e -c RT -p 20 command

This command would put the command into execution at a real-time priority of 120.

Process Scheduling Parameters

The following kernel parameters control aspects of process scheduling:

• maxclsyspri

  maxclsyspri is the maximum global priority of processes in the system class. When the kernel starts system processes, it assigns their priorities using the value of maxclsyspri as a reference point. maxclsyspri must have a value of 39 or greater, because the kernel assumes that the total range of system class priorities is at least 40.

  If you change this parameter, you must rebuild the scheduling class tables with values that correspond to the maximum priorities that you assign.

• sys_name

  sys_name is the character string name of the system scheduler class. The default value of sys_name is SYS.

Timesharing Policy

The following parameter is specified in the TS loadable module, which controls the timesharing policy:

• ts_maxupri

  ts_maxupri specifies the range within which users may adjust the priority of a timesharing process, using the priocntl(l) command or the priocntl(2) system call. The valid range for the user-supplied priority in the timesharing class is from +ts_maxupri to -ts_maxupri. The default value of ts_maxupri is 20 (which sets the range between +20 and -20, emulating the behavior of the older, less general scheduler interfaces, nice and setpriority.)

  The value of ts_maxupri is independent of the configured number of global timesharing priorities. In the default configuration, there are 0-59 timesharing priorities, but users may adjust their priorities only within a range of -20 to +20, relative to the system-calculated priority of the process. See "How to Designate a Process Priority" for more information.

  To change the value of this parameter, enter a line in /etc/system with the format:

  set TS:ts_maxupri=value
  

Real-Time Policy

The following parameter is specified in the RT loadable module, which controls the real-time policy:

• rt_maxpri

  rt_maxpri specifies the maximum priority to assign to real-time processes. The default value of rt_maxpri is 159.

  If you change this parameter, you must rebuild the scheduling class tables with values that correspond to the maximum priorities that you assign.

  To change the value of this parameter, enter a line in the /etc/system file with the format:

  set RT:rt_maxupri=value 
  

Scheduler Parameter Tables

The scheduler tables are described in Table 67-2.

These tables define scheduling policy by setting the scheduling parameters to use for real-time and timesharing processes. The parameters specify how much CPU time processes get at different priority levels.

Default time slices for the priority levels are specified in the ts_dptbl and rt_dptbl configuration tables, which are defined in the TS_DPTBL and RT_DPTBL loadable modules. These modules are automatically loaded from the /kernel/sched directory into the kernel as needed.

The time slices are specified in units (quanta) with a resolution defined by a "resolution" line. The default resolution is 1000, which means the time quantum values are interpreted as milliseconds. This is derived from the reciprocal of the specified resolution in seconds. The quanta are rounded up to the next integral multiple of the system clock's resolution in clock ticks. (The system clock ticks HZ times per second, where HZ is a hardware-dependent constant defined in the param.h header file.) For example, if the clock tick is 10 milliseconds, 42 quanta is rounded up to 50 milliseconds.

Timesharing Parameter Table

A default version of the ts_dptb, is delivered with the system in /kernel/sched/TS_DPTBL. The default configuration has 60 timesharing priorities.

The dispadmin -c TS -g command displays a sample ts_dptbl table.

$ dispadmin -c TS -g
# Time Sharing Dispatcher Configuration
RES=1000
 
# ts_quantum  ts_tqexp  ts_slpret  ts_maxwait ts_lwait  PRIORITY LEVEL
       200         0        50           0        50        #     0
       200         0        50           0        50        #     1
       200         0        50           0        50        #     2
       200         0        50           0        50        #     3
       200         0        50           0        50        #     4
       200         0        50           0        50        #     5
       200         0        50           0        50        #     6
       200         0        50           0        50        #     7
       200         0        50           0        50        #     8
       200         0        50           0        50        #     9
       160         0        51           0        51        #    10
       160         1        51           0        51        #    11
       160         2        51           0        51        #    12
       160         3        51           0        51        #    13
       160         4        51           0        51        #    14
       160         5        51           0        51        #    15
       160         6        51           0        51        #    16
       160         7        51           0        51        #    17
       160         8        51           0        51        #    18
       160         9        51           0        51        #    19
       120        10        52           0        52        #    20
       120        11        52           0        52        #    21
       120        12        52           0        52        #    22
       120        13        52           0        52        #    23
       120        14        52           0        52        #    24
       120        15        52           0        52        #    25
       120        16        52           0        52        #    26
       120        17        52           0        52        #    27
       120        18        52           0        52        #    28
       120        19        52           0        52        #    29
        80        20        53           0        53        #    30
        80        21        53           0        53        #    31
        80        22        53           0        53        #    32
        80        23        53           0        53        #    33
        80        24        53           0        53        #    34
        80        25        54           0        54        #    35
        80        26        54           0        54        #    36
        80        27        54           0        54        #    37
        80        28        54           0        54        #    38
        80        29        54           0        54        #    39
        40        30        55           0        55        #    40
        40        31        55           0        55        #    41
        40        32        55           0        55        #    42
        40        33        55           0        55        #    43
        40        34        55           0        55        #    44
        40        35        56           0        56        #    45
        40        36        57           0        57        #    46
        40        37        58           0        58        #    47
        40        38        58           0        58        #    48
        40        39        58           0        59        #    49
        40        40        58           0        59        #    50
        40        41        58           0        59        #    51
        40        42        58           0        59        #    52
        40        43        58           0        59        #    53
        40        44        58           0        59        #    54
        40        45        58           0        59        #    55
        40        46        58           0        59        #    56
        40        47        58           0        59        #    57
        40        48        58           0        59        #    58
        20        49        59       32000        59        #    59
$ 

Table 67-3 describes the fields in the ts_dptbl table.

Real-Time Parameter Table

A default version of rt_dptbl is delivered with the system in the /kernel/sched/RT_DPTBL loadable module.

The dispadmin -c RT -g command displays rt_dptbl information similar to the following.

$ dispadmin -c RT -g
# Real Time Dispatcher Configuration
RES=1000
 
# TIME QUANTUM                    PRIORITY
# (rt_quantum)                      LEVEL
      1000                    #        0
      1000                    #        1
      1000                    #        2
      1000                    #        3
      1000                    #        4
      1000                    #        5
      1000                    #        6
      1000                    #        7
      1000                    #        8
      1000                    #        9
       800                    #       10
       800                    #       11
       800                    #       12
       800                    #       13
       800                    #       14
       800                    #       15
       800                    #       16
       800                    #       17
       800                    #       18
       800                    #       19
       600                    #       20
       600                    #       21
       600                    #       22
       600                    #       23
       600                    #       24
       600                    #       25
       600                    #       26
       600                    #       27
       600                    #       28
       600                    #       29
       400                    #       30
       400                    #       31
       400                    #       32
       400                    #       33
       400                    #       34
       400                    #       35
       400                    #       36
       400                    #       37
       400                    #       38
       400                    #       39
       200                    #       40
       200                    #       41
       200                    #       42
       200                    #       43
       200                    #       44
       200                    #       45
       200                    #       46
       200                    #       47
       200                    #       48
       200                    #       49
       100                    #       50
       100                    #       51
       100                    #       52
       100                    #       53
       100                    #       54
       100                    #       55
       100                    #       56
       100                    #       57
       100                    #       58
       100                    #       59
$
 

Table 67-4 describes the fields in the real-time parameter table.

Kernel-Mode Parameter Table

The scheduler uses the kernel-mode parameter table, ts_kmdpris, to manage sleeping timesharing processes. A default version of ts_kmdpris is delivered with the system, in the /kernel/sched/TS_DPTBL loadable module, and is automatically built into the kernel as part of system configuration. See ts_dptbl(4) for more information.

The kernel assumes that it has at least 40 priorities in ts_kmdpris. It panics if it does not.

The kernel-mode parameter table is a one-dimensional array of global priorities from 60 through 99. If a process owns a critical resource, it is assigned a kernel priority so that it can release the resource as soon as possible. Critical resources are:

• An exclusive lock on a page

• A read lock on a readers/writer lock

Prior to SunOS 5.3, processes were assigned kernel priorities while they were asleep. This ensured that the resources they were waiting for were not paged out before they had a chance to execute again.

In order to do this after SunOS 5.3, processes return from sleep with the highest time-share priority (59).