Chapter 8 Physical Memory Management Using the Resource Capping Daemon

This feature is only available if you are using Solaris Resource Manager 1.3 with Solaris 8 SPARC Platform Edition.

A resource cap is an upper bound placed on the consumption of a resource, such as physical memory or CPU time, by a collection of processes.

In Solaris Resource Manager 1.3, resource cap enforcement enables you to regulate the resource consumption of physical memory by collections of processes attached to an lnode or to a project. For information about projects, see project(4).

Attribute for Limiting Physical Memory Usage

A physical memory resource cap can be defined for an lnode or a project (depending on the mode in use) by adding an attribute to the lnode or project.

The lnode attribute used to assign a resident set size (RSS) cap is:

rss.limit

The rss.limit attribute defines the physical memory cap, in specified units, that is applied to collections of processes attached to an lnode. The cap must be a positive number.

rss.limit is set through limadm(1MSRM). For example, to set a 10-gigabyte RSS cap for the lnode oracle, type:

# /usr/srm/sbin/limadm set rss.limit=10G oracle

The project attribute used to establish an RSS cap is:

rcap.max-rss

The rcap.max-rss attribute defines the total amount of physical memory, in bytes, that is available to processes attached to a project. The cap must be a positive number.

rcap.max-rss is configured in the project(4) database. For example, the following line in the project database sets an RSS cap for a project named oracle.

oracle:100::oracle,root::rcap.max-rss=10737418240

Technical Description

The resource cap enforcement daemon rcapd(1MSRM) and its associated utilities provide mechanisms for resource cap enforcement and administration. Only one instance of rcapd can run at any given time. The daemon periodically samples the resource utilization of collections of processes that have physical memory caps defined. If process consumption exceeds the cap quantity and other conditions are met, the daemon takes actions to reduce the processes' aggregated resource consumption to a level below the cap quantity.

The virtual memory system divides physical memory into segments known as pages. To read data from a file into memory, the virtual memory system reads in one page at a time, or pages in a file. To reduce resource consumption, the daemon can page out, or relocate, infrequently used pages to an area outside of physical memory.

Use rcapadm(1MSRM) to configure the resource capping daemon. rcapadm is also used to administer resource caps at a global level. Configuration changes can be incorporated into rcapd on demand by sending it SIGHUP (see kill(1)), or through the configuration interval (see rcapadm(1MSRM)). You must have superuser privileges to configure the daemon. If used without arguments, rcapadm displays the current status of the resource capping daemon if it has been configured.

Use rcapstat(1SRM) to report resource cap enforcement daemon statistics for each cap defined.

Using the Resource Capping Daemon

Use rcapadm(1MSRM) to configure the daemon.

Selecting the Mode of Operation

Two modes of operation are supported, lnode and project.

To enforce caps defined for lnodes and ignore caps defined for projects, type:

# rcapadm -m lnode

To enforce caps defined for projects and ignore caps defined for lnodes, type:

# rcapadm -m project

Tuning Operation Intervals

You can tune the intervals for the periodic operations performed by rcapd. To reset intervals, use the -i option.

# rcapadm -i interval=value,...,interval=value 

All intervals are specified in seconds. The intervals and their default values are described in the following table.

If the interval specified to rcapstat is shorter than the interval specified to rcapd (with rcapadm(1MSRM)), the output for some intervals can be zero. This is because rcapd does not update statistics more frequently than the interval specified with rcapadm, and this interval is independent of, and less precise than, the sampling interval used by rcapstat.

Setting Memory Cap Enforcement Value

Caps can be configured so that they will not be enforced until the physical memory available to processes is low. The minimum (and default) value is 0, which means that memory caps are always enforced. To set a different minimum physical memory utilization for memory cap enforcement, type:

# rcapadm -c percent

The percent value should be in the range 0 to 100. The current global physical memory utilization and its cap can be obtained by using the -g option to the rcapstat command. See Reporting Global Memory Caps.

Enabling Resource Capping

To enable the resource capping daemon so that it will be started now and also be started each time the system is booted, type :

# rcapadm -E

To enable the resource capping daemon at boot without affecting its current running status, also specify the -n option:

# rcapadm -n -E

Disabling Resource Capping

To disable the the resource capping daemon so that it will be stopped now and not be started when the system is booted, type :

# rcapadm -D

To disable the resource capping daemon without affecting its current running status, also specify the -n option:

# rcapadm -n -D

If killed, rcapd can leave processes in a stopped state. See kill(1). Use rcapadm -D or SIGTERM to cause rcapd to terminate properly.

Monitoring the Resource Capping Daemon

Use the rcapstat command to issue reports on lnodes or projects capped by rcapd. You can set the sampling interval and specify the report frequency.

interval

Specifies the sampling interval in seconds. The default interval is 5 seconds.

count

Specifies the number of times that the statistics are repeated. By default, rcapstat reports statistics until a termination signal is received or until the rcapd process exits.

The paging statistics in the first report issued by rcapstat show the activity since the daemon was started. Subsequent reports reflect the activity since the last report was issued.

The following list defines the column headings in an rcapstat report.

id

The lnode ID of the collection of processes or project ID of the processes attached to a project.

lnode/project

The collection ID type, which is project or lnode.

nproc

The number of processes in the collection.

vm

The total virtual memory size of the collection of processes, including all mapped files and devices, in kilobytes (K), megabytes (M), or gigabytes (G).

rss

The total resident set size (RSS) of the collection of processes, in kilobytes (K), megabytes (M), or gigabytes (G), not accounting for objects that are shared.

cap

The RSS cap defined for a project or an lnode. See rcapd(1MSRM) for information about how to specify memory caps.

at

The total amount of memory that rcapd attempted to page out since the last rcapstat sample.

avgat

The average amount of memory that rcapd attempted to page out during each sample cycle that occurred since the last rcapstat sample. The rate at which rcapd samples collection RSS can be set with rcapadm(1MSRM).

pg

The total amount of memory that rcapd successfully paged out since the last rcapstat sample. rcapd pages out the most infrequently used pages.

avgpg

An estimate of the average amount of memory that rcapd successfully paged out during each sample cycle that occurred since the last rcapstat sample. The rate at which rcapd samples process RSS sizes can be set with rcapadm.

Producing Reports With rcapstat

The examples in this section show you how to use rcapstat to monitor the resource utilization of collections of processes that have physical memory caps defined.

Using rcapstat to Report Cap and Lnode Information

Caps are defined for two lnodes associated with two users. user1 has a cap of 50 megabytes, and user2 has a cap of 10 megabytes.

The following command produces reports at 5-second sampling intervals. A report will be issued five times, once after each sample.

usermachine% rcapstat 5 5
    id lnode   nproc    vm   rss   cap    at avgat    pg avgpg
112270 user1      24  123M   35M   50M   50M    0K 3312K    0K
 78194 user2       1 2368K 1856K   10M    0K    0K    0K    0K
    id lnode   nproc    vm   rss   cap    at avgat    pg avgpg
112270 user1      24  123M   35M   50M    0K    0K    0K    0K
 78194 user2       1 2368K 1856K   10M    0K    0K    0K    0K
    id lnode   nproc    vm   rss   cap    at avgat    pg avgpg
112270 user1      24  123M   35M   50M    0K    0K    0K    0K
 78194 user2       1 2368K 1928K   10M    0K    0K    0K    0K
    id lnode   nproc    vm   rss   cap    at avgat    pg avgpg
112270 user1      24  123M   35M   50M    0K    0K    0K    0K
 78194 user2       1 2368K 1928K   10M    0K    0K    0K    0K
    id lnode   nproc    vm   rss   cap    at avgat    pg avgpg
112270 user1      24  123M   35M   50M    0K    0K    0K    0K
 78194 user2       1 2368K 1928K   10M    0K    0K    0K    0K

The first three lines of output constitute the first report, which contains the cap and lnode information for the two lnodes, and paging statistics since rcapd was started. The at and pg columns are a number greater than zero for user1 and zero for user2, which indicates that at some time in the daemon's history, user1 exceeded its cap but user2 did not.

The subsequent reports contain paging statistics since the prior intervals, but show no significant activity.

Lowering the Cap of an Lnode

The limadm(1MSRM) command can be used to lower the memory cap of an lnode, which makes the cap more restrictive. rcapd will enforce the new cap after the next configuration interval (see rcapadm(1MSRM)). A signal can also be sent, which causes rcapd to enforce the new cap immediately.

admin# limadm set rss.limit=30M user1
admin# pkill -HUP rcapd

The following command produces reports at 5-second sampling intervals. A report will be issued five times, once after each sample.

admin# rcapstat 5 5
    id lnode   nproc    vm   rss   cap    at avgat    pg avgpg
112270 user1      24  123M   35M   30M   50M    0K 3312K    0K
 78194 user2       1 2368K 1856K   10M    0K    0K    0K    0K
    id lnode   nproc    vm   rss   cap    at avgat    pg avgpg
112270 user1      24  123M   36M   30M   52M   52M  632K  632K
 78194 user2       1 2368K 2096K   10M    0K    0K    0K    0K
    id lnode   nproc    vm   rss   cap    at avgat    pg avgpg
112270 user1      24  123M   33M   30M   57M   52M  816K  632K
 78194 user2       1 2368K 1968K   10M    0K    0K    0K    0K
    id lnode   nproc    vm   rss   cap    at avgat    pg avgpg
112270 user1      24  123M   27M   30M 4792K 4792K   40K   40K
 78194 user2       1 2368K 1144K   10M    0K    0K    0K    0K
    id lnode   nproc    vm   rss   cap    at avgat    pg avgpg
112270 user1      24  123M   27M   30M    0K    0K    0K    0K
 78194 user2       1 2368K 1144K   10M    0K    0K    0K    0K

When the cap was lowered to 30 megabytes from 50 megabytes, rcapd responded by attempting to page out the 6-megabyte amount of resident memory above the cap value. The goal was reached; it was exceeded by a small amount.

Using rcapstat to Monitor the RSS of a Project

The following command produces reports at 5-second sampling intervals. A report will be issued five times, once after each sample.

user1machine% rcapstat 5 5

id     project   nproc    vm   rss   cap    at avgat    pg avgpg
376565 user1        57  209M   46M   10M  440M  220M 5528K 2764K
376565 user1        57  209M   44M   10M  394M  131M 4912K 1637K
376565 user1        56  207M   43M   10M  440M  147M 6048K 2016K
376565 user1        56  207M   42M   10M  522M  174M 4368K 1456K
376565 user1        56  207M   44M   10M  482M  161M 3376K 1125K

In this example, the project user1 has an RSS in excess of its physical memory cap. The nonzero values in the pg column indicate that rcapd is consistently paging out memory as it attempts to meet the cap by lowering the physical memory utilization of the project's processes. However, rcapd is unsuccessful, as indicated by the varying rss values that do not show a corresponding decrease. This means that the application's resident memory is being actively used, forcing rcapd to affect the working set. Under this condition, the system will continue to experience high page fault rates, and associated I/O, until the working set size is reduced, the cap is raised, or the application changes its memory access pattern.

Determining the Working Set Size of a Project

The following example is a continuation of the previous example, and it uses the same project.

example% rcapstat 5 5
    id project    nproc    vm   rss   cap    at avgat    pg avgpg
376565 user1         56  207M   44M   10M  381M  191M   15M 7924K
376565 user1         56  207M   46M   10M  479M  160M 2696K  898K
376565 user1         56  207M   46M   10M  424M  141M 7280K 2426K
376565 user1         56  207M   43M   10M  401M  201M 4808K 2404K
376565 user1         56  207M   43M   10M  456M  152M 4800K 1600K
376565 user1         56  207M   44M   10M  486M  162M 4064K 1354K
376565 user1         56  207M   52M  100M  191M   95M 1944K  972K
376565 user1         56  207M   55M  100M    0K    0K    0K    0K
376565 user1         56  207M   56M  100M    0K    0K    0K    0K
376565 user1         56  207M   56M  100M    0K    0K    0K    0K
376565 user1         56  207M   56M  100M    0K    0K    0K    0K
376565 user1         56  207M   56M  100M    0K    0K    0K    0K

By inhibiting cap enforcement, either by raising the cap of a project or by changing the minimum cap enforcement memory pressure value (see rcapadm(1MSRM)), the resident set can become the working set. The rss column might stabilize to show the project working set size, as shown in this example. This is the minimum cap value that will allow the project's processes to operate without perpetually incurring page faults.

Reporting Global Memory Caps

You can use the -g option of rcapstat to report the following:

• Global memory utilization cap set by rcapadm(1MSRM)

• Current memory utilization as a percentage of all physical memory installed on the system

References

For additional information, see rcapadm(1MSRM), rcapd(1MSRM), rcapstat(1SRM), limadm(1MSRM), and project(4).