Chapter 8 Resource Controls

After you determine the resource consumption of workloads on your system as described in Chapter 7, Extended Accounting, you can place bounds on resource usage. Bounds prevent workloads from over-consuming resources. The resource controls facility, which extends the UNIX resource limit concept, is the constraint mechanism that is used for this purpose.

Overview

UNIX systems have traditionally provided a resource limit facility (rlimit). The rlimit facility allows administrators to set one or more numerical limits on the amount of resources a process can consume. These limits include per-process CPU time used, per-process core file size, and per-process maximum heap size. Heap size is the amount of memory that is allocated for the process data segment.

In the Solaris operating environment, the concept of a per-process resource limit has been extended to the task and project entities described in Chapter 6, Projects and Tasks. The extended limits can be observed on a system-wide basis. These enhancements are provided by the resource controls (rctls) facility.

The resource controls facility provides compatibility interfaces for the resource limits facility. Existing applications that use resource limits continue to run unchanged. These applications can be observed in the same way as applications that are modified to take advantage of the resource controls facility.

Resource controls provide a mechanism for constraint on system resources. Processes, tasks, and projects can be prevented from consuming amounts of specified system resources. This mechanism leads to a more manageable system by preventing over-consumption of resources.

Constraint mechanisms can be used to support capacity-planning processes. An encountered constraint can provide information about application resource needs without necessarily denying the resource to the application.

Resource controls can also serve as a simple attribute mechanism for resource management facilities. For example, the number of CPU shares made available to a project in the fair share scheduler (FSS) scheduling class is defined by the project.cpu-shares resource control. Because a project is assigned a fixed number of shares by the control, the various actions associated with exceeding a control are not relevant. In this context, the current value for the project.cpu-shares control is considered an attribute on the specified project.

Administering Resource Controls

The resource controls facility is configured through the project database (see Chapter 6, Projects and Tasks). Resource control attributes are set in the final field of the project database entry. The values associated with each resource control are enclosed in parentheses, and appear as plain text separated by commas. The values in parentheses constitute an "action clause." Each action clause is composed of a privilege level, a threshold value, and an action that is associated with the particular threshold. Each resource control can have multiple action clauses, which are also separated by commas. The following entry defines a per-process address-space limit and a per-task lightweight process limit on a project entity.

development:101:Developers:::task.max-lwps=(privileged,10,deny);
  process.max-address-space=(privileged,209715200,deny)

The rctladm(1M) command allows you to make runtime interrogations of and modifications to the resource controls facility, with global scope. The prctl(1) command allows you to make runtime interrogations of and modifications to the resource controls facility, with local scope.

Available Resource Controls

A list of the standard resource controls that are available in this release is shown in the following table.

The table describes the resource that is constrained by each control. The table also identifies the default units that are used by the project database for that resource. The default units are of two types:

• Quantities represent a limited amount.

• Indexes represent a maximum valid identifier.

Thus, project.cpu-shares specifies the number of shares to which the project is entitled. process.max-file-descriptor specifies the highest file number that can be assigned to a process by the open(2) system call.

Resource Control Values and Privilege Levels

A threshold value on a resource control constitutes an enforcement point where local actions can be triggered or global actions, such as logging, can occur.

Each threshold value must be associated with a privilege level of one of the following three types.

• Basic, which can be modified by the owner of the calling process

• Privileged, which can be modified only by privileged (superuser) callers

• System, which is fixed for the duration of the operating system instance

A resource control is guaranteed to have one system value, which is defined by the system, or resource provider. The system value represents how much of the resource the current implementation of the operating system is capable of providing.

Any number of privileged values can be defined, and only one basic value is allowed. Operations that are performed without specifying a privilege value are assigned a basic privilege by default.

The privilege level for a resource control value is defined in the privilege field of the resource control block as RCTL_BASIC, RCTL_PRIVILEGED, or RCTL_SYSTEM. See getrctl(2) for more information. You can use the prctl command to modify values that are associated with basic and privileged levels.

Actions on Resource Control Values

For each threshold value that is placed on a resource control, you can associate one or more actions.

• You can choose to deny the resource requests for an amount that is greater than the threshold.

• You can choose to send a signal to the violating or observing process if the threshold value is reached.

Due to implementation restrictions, the global properties of each control can restrict the set of available actions that can be set on the threshold value. A list of available signal actions is presented in the following table. For additional information on signals, see signal(3HEAD).

Resource Control Flags and Properties

Each resource control on the system has a certain set of associated properties. This set of properties is defined as a set of global flags, which are associated with all controlled instances of that resource. Global flags cannot be modified, but the flags can be retrieved by using either rctladm or the getrctl system call.

Local flags define the default behavior and configuration for a specific threshold value of that resource control on a specific process or process collective. The local flags for one threshold value do not affect the behavior of other defined threshold values for the same resource control. However, the global flags affect the behavior for every value associated with a particular control. Local flags can be modified, within the constraints supplied by their corresponding global flags, by the prctl command or the setrctl system call (see setrctl(2)).

For the complete list of local flags, global flags, and their definitions, see rctlblk_set_value(3C).

To determine system behavior when a threshold value for a particular resource control is reached, use rctladm to display the global flags for the resource control . For example, to display the values for process.max-cpu-time, type the following:

$ rctladm process.max-cpu-time
	process.max-cpu-time   syslog=off   [ lowerable no-deny cpu-time inf ]

The global flags indicate the following.

lowerable

Superuser privileges are not required to lower the privileged values for this control.

no-deny

Even when threshold values are exceeded, access to the resource is never denied.

cpu-time

SIGXCPU is available to be sent when threshold values of this resource are reached.

inf

Any value with RCTL_LOCAL_MAXIMAL defined actually represents an infinite quantity, and the value is never enforced.

Use prctl to display local values and actions for the resource control.

$ prctl -n process.max-cpu-time $$
	353939: -ksh
	process.max-cpu-time   [ lowerable no-deny cpu-time inf ]
		18446744073709551615 privileged signal=XCPU   [ max ]
		18446744073709551615 system     deny          [ max ]

The max (RCTL_LOCAL_MAXIMAL) flag is set for both threshold values, and the inf (RCTL_GLOBAL_INFINITE) flag is defined for this resource control. Hence, as configured, both threshold quantities represent infinite values and they will never be exceeded.

Resource Control Enforcement

More than one resource control can exist on a resource. A resource control can exist at each containment level in the process model. If resource controls are active on the same resource at different container levels, the smallest container's control is enforced first. Thus, action is taken on process.max-cpu-time before task.max-cpu-time if both controls are encountered simultaneously.

Figure 8-1 Process Collectives, Container Relationships, and Their Resource Control Sets

Global Monitoring of Resource Control Events

Often, the resource consumption of processes is unknown. To get more information, try using the global resource control actions that are available with rctladm(1M). Use rctladm to establish a syslog action on a resource control. Then, if any entity managed by that resource control encounters a threshold value, a system message is logged at the configured logging level.

Configuration

Each resource control listed in Table 8-1 can be assigned to a project at login or when newtask(1) or the other project-aware launchers at(1), batch (see at(1)), or cron(1M) are invoked. Each command that is initiated is launched in a separate task with the invoking user's default project.

Updates to entries in the project database, whether to the /etc/project file or to a representation of the database in a network name service, are not applied to currently active projects. The updates are applied when a new task joins the project through login(1) or newtask.

Temporarily Updating Resource Control Values on a Running System

Values changed in the project database only become effective for new tasks that are started in a project. However, you can use the rctladm and prctl commands to update resource controls on a running system.

Updating Logging Status

The rctladm command affects the global logging state of each resource control on a system-wide basis. This command can be used to view the global state and to set up the level of syslog logging when controls are exceeded.

Updating Resource Controls

You can view and temporarily alter resource control values and actions on a per-process, per-task, or per-project basis by using prctl. A project, task, or process ID is given as input, and the command operates on the resource control at the level where it is defined.

Any modifications to values and actions take effect immediately. However, these modifications apply to the current session only. The changes are not recorded in the project database. If the system is restarted, the modifications are lost. Permanent changes to resource controls must be made in the project database.

All resource control settings that can be modified in the project database can also be modified with the prctl command. Both basic and privileged values can be added or be deleted and their actions can be modified. By default, the basic type is assumed for all set operations, but processes and users with superuser privileges can also modify privileged resource controls. System resource controls cannot be altered.

Using Resource Controls

How to Set the Maximum Number of LWPs for Each Task in a Project

Type this entry in the /etc/project database to set the maximum number of LWPs in each task in project x-files to 3.

x-files:100::root::task.max-lwps=(privileged,3,deny)

When superuser creates a new task in project x-files by joining it with newtask, superuser will not be able to create more than three LWPs while running in this task. This is shown in the following annotated sample session.

# newtask -p x-files csh

# prctl -n task.max-lwps $$
688: csh
task.max-lwps
                            3 privileged deny
                   2147483647 system     deny
# id -p
uid=0(root) gid=1(other) projid=100(x-files)

# ps -o project,taskid -p $$
 PROJECT TASKID
 x-files   236

# csh        /* creates second LWP */

# csh        /* creates third LWP */

# csh        /* cannot create more LWPs */
Vfork failed

#

How to Set Multiple Controls on a Project

The /etc/project file can contain settings for multiple resource controls for each project as well as multiple threshold values for each control. Threshold values are defined in action clauses, which are comma-separated for multiple values.

The following line in the file sets a basic control with no action on the maximum LWPs per task for project x-files. The line also sets a privileged deny control on the maximum LWPs per task. This control causes any LWP creation that exceeds the maximum to fail, as shown in the previous example. Finally, the maximum file descriptors per process are limited at the basic level, which forces failure of any open call that exceeds the maximum.

x-files:101::root::task.max-lwps=(basic,10,none),(privileged,500,deny);
    process.max-file-descriptor=(basic,128,deny)

How to Use prctl

As superuser, type prctl to display the maximum file descriptor for the current shell that is running:

# prctl -n process.max-file-descriptor $$
8437:   sh
process.max-file-descriptor              [ lowerable deny ]
                          256 basic      deny
                        65536 privileged deny
                   2147483647 system     deny

Use the prctl command to temporarily add a new privileged value to deny the use of more than three LWPs per task for the x-files project. The result is identical to the result in "How to Set the Maximum Number of LWPs for Each Task in a Project", as shown in the following annotated sample session:

# newtask -p x-files

# id -p
uid=0(root) gid=1(other) projid=101(x-files)

# prctl -n task.max-lwps -t privileged -v 3 -e deny -i project x-files

# prctl -n task.max-lwps -i project x-files
670:    sh
task.max-lwps
                            3 privileged deny
                   2147483647 system     deny

You can also use prctl -r to change the lowest value of a resource control.

# prctl -n process.max-file-descriptor -r -v 128 $$

How to Use rctladm

You can use rctladm to enable the global syslog attribute of a resource control. When the control is exceeded, notification is logged at the specified syslog level. Type the following:

# rctladm -e syslog process.max-file-descriptor

Capacity Warnings

A global action on a resource control enables you to receive notice of any entity that is tripping over a resource control value.

For example, assume you want to determine whether a web server possesses sufficient CPUs for its typical workload. You could analyze sar(1) data for idle CPU time and load average. You could also examine extended accounting data to determine the number of simultaneous processes that are running for the web server process.

However, an easier approach is to place the web server in a task. You can then set a global action, using syslog, to notify you whenever a task exceeds a scheduled number of LWPs appropriate for the machine's capabilities.

How to Determine Whether a Web Server Is Allocated Enough CPU Capacity

1. Use the prctl command to place a privileged (superuser-owned) resource control on the tasks that contain an httpd process. Limit each task's total number of LWPs to 40, and disable all local actions.

   # prctl -n task.max-lwps -v 40 -t privileged -d all `pgrep httpd`

2. Enable a system log global action on the task.max-lwps resource control.

   # rctladm -e syslog task.max-lwps

3. Observe whether the workload trips the resource control.

   If it does, you will see /var/adm/messages such as:

   Jan 8 10:15:15 testmachine unix: [ID 859581 kern.notice] NOTICE: privileged rctl task.max-lwps exceeded by task 19