System Administration Guide: Virtualization Using the Solaris Operating System

Chapter 12 Resource Pools (Overview)

This chapter discusses the following features:

Resource pools and dynamic resource pools are services in the Solaris service management facility (SMF). Each of these services is enabled separately.

The following topics are covered in this chapter:

For procedures using this functionality, see Chapter 13, Creating and Administering Resource Pools (Tasks).

Introduction to Resource Pools

Resource pools enable you to separate workloads so that workload consumption of certain resources does not overlap. This resource reservation helps to achieve predictable performance on systems with mixed workloads.

Resource pools provide a persistent configuration mechanism for processor set (pset) configuration and, optionally, scheduling class assignment.

Figure 12–1 Resource Pool Framework

Illustration shows that a pool is made up of one processor
set and optionally, a scheduling class.

A pool can be thought of as a specific binding of the various resource sets that are available on your system. You can create pools that represent different kinds of possible resource combinations:

pool1: pset_default

pool2: pset1

pool3: pset1, pool.scheduler="FSS"

By grouping multiple partitions, pools provide a handle to associate with labeled workloads. Each project entry in the /etc/project file can have a single pool associated with that entry, which is specified using the project.pool attribute.

When pools are enabled, a default pool and a default processor set form the base configuration. Additional user-defined pools and processor sets can be created and added to the configuration. A CPU can only belong to one processor set. User-defined pools and processor sets can be destroyed. The default pool and the default processor set cannot be destroyed.

The default pool has the pool.default property set to true. The default processor set has the pset.default property set to true. Thus, both the default pool and the default processor set can be identified even if their names have been changed.

The user-defined pools mechanism is primarily for use on large machines of more than four CPUs. However, small machines can still benefit from this functionality. On small machines, you can create pools that share noncritical resource partitions. The pools are separated only on the basis of critical resources.

Introduction to Dynamic Resource Pools

Dynamic resource pools provide a mechanism for dynamically adjusting each pool's resource allocation in response to system events and application load changes. DRPs simplify and reduce the number of decisions required from an administrator. Adjustments are automatically made to preserve the system performance goals specified by an administrator. The changes made to the configuration are logged. These features are primarily enacted through the resource controller poold, a system daemon that should always be active when dynamic resource allocation is required. Periodically, poold examines the load on the system and determines whether intervention is required to enable the system to maintain optimal performance with respect to resource consumption. The poold configuration is held in the libpool configuration. For more information on poold, see the poold(1M) man page.

About Enabling and Disabling Resource Pools and Dynamic Resource Pools

To enable and disable resource pools and dynamic resource pools, see Enabling and Disabling the Pools Facility.

Resource Pools Used in Zones

As an alternative to associating a zone with a configured resource pool on your system, you can use the zonecfg command to create a temporary pool that is in effect while the zone is running. See dedicated-cpu Resource for more information.

On a system that has zones enabled, a non-global zone can be associated with one resource pool, although the pool need not be exclusively assigned to a particular zone. Moreover, you cannot bind individual processes in non-global zones to a different pool by using the poolbind command from the global zone. To associate a non-global zone with a pool, see Configuring, Verifying, and Committing a Zone.

Note that if you set a scheduling class for a pool and you associate a non-global zone with that pool, the zone uses that scheduling class by default.

If you are using dynamic resource pools, the scope of an executing instance of poold is limited to the global zone.

The poolstat utility run in a non-global zone displays only information about the pool associated with the zone. The pooladm command run without arguments in a non-global zone displays only information about the pool associated with the zone.

For information about resource pool commands, see Commands Used With the Resource Pools Facility.

When to Use Pools

Resource pools offer a versatile mechanism that can be applied to many administrative scenarios.

Batch compute server

Use pools functionality to split a server into two pools. One pool is used for login sessions and interactive work by timesharing users. The other pool is used for jobs that are submitted through the batch system.

Application or database server

Partition the resources for interactive applications in accordance with the applications' requirements.

Turning on applications in phases

Set user expectations.

You might initially deploy a machine that is running only a fraction of the services that the machine is ultimately expected to deliver. User difficulties can occur if reservation-based resource management mechanisms are not established when the machine comes online.

For example, the fair share scheduler optimizes CPU utilization. The response times for a machine that is running only one application can be misleadingly fast. Users will not see these response times with multiple applications loaded. By using separate pools for each application, you can place a ceiling on the number of CPUs available to each application before you deploy all applications.

Complex timesharing server

Partition a server that supports large user populations. Server partitioning provides an isolation mechanism that leads to a more predictable per-user response.

By dividing users into groups that bind to separate pools, and using the fair share scheduling (FSS) facility, you can tune CPU allocations to favor sets of users that have priority. This assignment can be based on user role, accounting chargeback, and so forth.

Workloads that change seasonally

Use resource pools to adjust to changing demand.

Your site might experience predictable shifts in workload demand over long periods of time, such as monthly, quarterly, or annual cycles. If your site experiences these shifts, you can alternate between multiple pools configurations by invoking pooladm from a cron job. (See Resource Pools Framework.)

Real-time applications

Create a real-time pool by using the RT scheduler and designated processor resources.

System utilization

Enforce system goals that you establish.

Use the automated pools daemon feature to identify available resources and then monitor workloads to detect when your specified objectives are no longer being satisfied. The daemon can take corrective action if possible, or the condition can be logged.

Resource Pools Framework

The /etc/pooladm.conf configuration file describes the static pools configuration. A static configuration represents the way in which an administrator would like a system to be configured with respect to resource pools functionality. An alternate file name can be specified.

When the service management facility (SMF) or the pooladm -e command is used to enable the resource pools framework, then, if an /etc/pooladm.conf file exists, the configuration contained in the file is applied to the system.

The kernel holds information about the disposition of resources within the resource pools framework. This is known as the dynamic configuration, and it represents the resource pools functionality for a particular system at a point in time. The dynamic configuration can be viewed by using the pooladm command. Note that the order in which properties are displayed for pools and resource sets can vary. Modifications to the dynamic configuration are made in the following ways:

More than one static pools configuration file can exist, for activation at different times. You can alternate between multiple pools configurations by invoking pooladm from a cron job. See the cron(1M) man page for more information on the cron utility.

By default, the resource pools framework is not active. Resource pools must be enabled to create or modify the dynamic configuration. Static configuration files can be manipulated with the poolcfg or libpool commands even if the resource pools framework is disabled. Static configuration files cannot be created if the pools facility is not active. For more information on the configuration file, see Creating Pools Configurations.

The commands used with resource pools and the poold system daemon are described in the following man pages:

/etc/pooladm.conf Contents

All resource pool configurations, including the dynamic configuration, can contain the following elements.


Properties affecting the total behavior of the system


A resource pool definition


A processor set definition


A processor definition

All of these elements have properties that can be manipulated to alter the state and behavior of the resource pools framework. For example, the pool property pool.importance indicates the relative importance of a given pool. This property is used for possible resource dispute resolution. For more information, see libpool(3LIB).

Pools Properties

The pools facility supports named, typed properties that can be placed on a pool, resource, or component. Administrators can store additional properties on the various pool elements. A property namespace similar to the project attribute is used.

For example, the following comment indicates that a given pset is associated with a particular Datatree database.


For additional information about property types, see poold Properties.

Note –

A number of special properties are reserved for internal use and cannot be set or removed. See the libpool(3LIB) man page for more information.

Implementing Pools on a System

User-defined pools can be implemented on a system by using one of these methods.

For information about enabling and disabling resource pools, see Enabling and Disabling the Pools Facility. The pools facility cannot be disabled when there are user-defined pools or resources in use.

To configure resource pools, you must have superuser privileges or have the Process Management profile in your list of profiles. The System Administrator role includes the Process Management profile.

The poold resource controller is started with the dynamic resource pools facility.

project.pool Attribute

The project.pool attribute can be added to a project entry in the /etc/project file to associate a single pool with that entry. New work that is started on a project is bound to the appropriate pool. See Chapter 2, Projects and Tasks (Overview) for more information.

For example, you can use the projmod command to set the project.pool attribute for the project sales in the /etc/project file:

# projmod -a -K project.pool=mypool sales

SPARC: Dynamic Reconfiguration Operations and Resource Pools

Dynamic Reconfiguration (DR) enables you to reconfigure hardware while the system is running. A DR operation can increase, reduce, or have no effect on a given type of resource. Because DR can affect available resource amounts, the pools facility must be included in these operations. When a DR operation is initiated, the pools framework acts to validate the configuration.

If the DR operation can proceed without causing the current pools configuration to become invalid, then the private configuration file is updated. An invalid configuration is one that cannot be supported by the available resources.

If the DR operation would cause the pools configuration to be invalid, then the operation fails and you are notified by a message to the message log. If you want to force the configuration to completion, you must use the DR force option. The pools configuration is then modified to comply with the new resource configuration. For information on the DR process and the force option, see the dynamic reconfiguration user guide for your Sun hardware.

If you are using dynamic resource pools, note that it is possible for a partition to move out of poold control while the daemon is active. For more information, see Identifying a Resource Shortage.

Creating Pools Configurations

The configuration file contains a description of the pools to be created on the system. The file describes the elements that can be manipulated.

See poolcfg(1M) for more information on elements that be manipulated.

When pools are enabled, you can create a structured /etc/pooladm.conf file in two ways.

Use poolcfg or libpool to modify the /etc/pooladm.conf file. Do not directly edit this file.

Directly Manipulating the Dynamic Configuration

It is possible to directly manipulate CPU resource types in the dynamic configuration by using the poolcfg command with the -d option. There are two methods used to transfer resources.

For an example, see Transferring Resources.

If DRP is in use, note that the resource transfer might trigger action from poold. See poold Overview for more information.

poold Overview

The pools resource controller, poold, uses system targets and observable statistics to preserve the system performance goals that you specify. This system daemon should always be active when dynamic resource allocation is required.

The poold resource controller identifies available resources and then monitors workloads to determine when the system usage objectives are no longer being met. poold then considers alternative configurations in terms of the objectives, and remedial action is taken. If possible, the resources are reconfigured so that objectives can be met. If this action is not possible, the daemon logs that user-specified objectives can no longer be achieved. Following a reconfiguration, the daemon resumes monitoring workload objectives.

poold maintains a decision history that it can examine. The decision history is used to eliminate reconfigurations that historically did not show improvements.

Note that a reconfiguration can also be triggered asynchronously if the workload objectives are changed or if the resources available to the system are modified.

Managing Dynamic Resource Pools

The DRP service is managed by the service management facility (SMF) under the service identifier svc:/system/pools/dynamic.

Administrative actions on this service, such as enabling, disabling, or requesting restart, can be performed using the svcadm command. The service's status can be queried using the svcs command. See the svcs(1) andsvcadm(1M) man pages for more information.

The SMF interface is the preferred method for controlling DRP, but for backward compatibility, the following methods can also be used.

Configuration Constraints and Objectives

When making changes to a configuration, poold acts on directions that you provide. You specify these directions as a series of constraints and objectives. poold uses your specifications to determine the relative value of different configuration possibilities in relation to the existing configuration. poold then changes the resource assignments of the current configuration to generate new candidate configurations.

Configuration Constraints

Constraints affect the range of possible configurations by eliminating some of the potential changes that could be made to a configuration. The following constraints, which are specified in the libpool configuration, are available.

See the libpool(3LIB) man page and Pools Properties for more information about pools properties.

See How to Set Configuration Constraints for usage instructions.

pset.min Property and pset.max Property Constraints

These two properties place limits on the number of processors that can be allocated to a processor set, both minimum and maximum. See Table 12–1 for more details about these properties.

Within these constraints, a resource partition's resources are available to be allocated to other resource partitions in the same Solaris instance. Access to the resource is obtained by binding to a pool that is associated with the resource set. Binding is performed at login or manually by an administrator who has the PRIV_SYS_RES_CONFIG privilege.

cpu.pinned Property Constraint

The cpu-pinned property indicates that a particular CPU should not be moved by DRP from the processor set in which it is located. You can set this libpool property to maximize cache utilization for a particular application that is executing within a processor set.

See Table 12–1 for more details about this property.

pool.importance Property Constraint

The pool.importance property describes the relative importance of a pool as defined by the administrator.

Configuration Objectives

Objectives are specified similarly to constraints. The full set of objectives is documented in Table 12–1.

There are two categories of objectives.

Workload dependent

A workload-dependent objective is an objective that will vary according to the nature of the workload running on the system. An example is the utilization objective. The utilization figure for a resource set will vary according to the nature of the workload that is active in the set.

Workload independent

A workload-independent objective is an objective that does not vary according to the nature of the workload running on the system. An example is the CPU locality objective. The evaluated measure of locality for a resource set does not vary with the nature of the workload that is active in the set.

You can define three types of objectives.


Valid Elements 










loose | tight | none



< > ~


Objectives are stored in property strings in the libpool configuration. The property names are as follows:

Objectives have the following syntax:

All objectives take an optional importance prefix. The importance acts as a multiplier for the objective and thus increases the significance of its contribution to the objective function evaluation. The range is from 0 to INT64_MAX (9223372036854775807). If not specified, the default importance value is 1.

Some element types support more than one type of objective. An example is pset. You can specify multiple objective types for these elements. You can also specify multiple utilization objectives on a single pset element.

See How to Define Configuration Objectives for usage examples.

wt-load Objective

The wt-load objective favors configurations that match resource allocations to resource utilizations. A resource set that uses more resources will be given more resources when this objective is active. wt-load means weighted load.

Use this objective when you are satisfied with the constraints you have established using the minimum and maximum properties, and you would like the daemon to manipulate resources freely within those constraints.

The locality Objective

The locality objective influences the impact that locality, as measured by locality group (lgroup) data, has upon the selected configuration. An alternate definition for locality is latency. An lgroup describes CPU and memory resources. The lgroup is used by the Solaris system to determine the distance between resources, using time as the measurement. For more information on the locality group abstraction, see Locality Groups Overview in Programming Interfaces Guide.

This objective can take one of the following three values:


If set, configurations that maximize resource locality are favored.


If set, configurations that minimize resource locality are favored.


If set, the favorableness of a configuration is not influenced by resource locality. This is the default value for the locality objective.

In general, the locality objective should be set to tight. However, to maximize memory bandwidth or to minimize the impact of DR operations on a resource set, you could set this objective to loose or keep it at the default setting of none.

utilization Objective

The utilization objective favors configurations that allocate resources to partitions that are not meeting the specified utilization objective.

This objective is specified by using operators and values. The operators are as follows:


The “less than” operator indicates that the specified value represents a maximum target value.


The “greater than” operator indicates that the specified value represents a minimum target value.


The “about” operator indicates that the specified value is a target value about which some fluctuation is acceptable.

A pset can only have one utilization objective set for each type of operator.

You can set both a < and a > operator together to create a range. The values will be validated to make sure that they do not overlap.

Configuration Objectives Example

In the following example, poold is to assess these objectives for the pset:

Example 12–1 poold Objectives Example

pset.poold.objectives "utilization > 30; utilization < 80; locality tight"

See How to Define Configuration Objectives for additional usage examples.

poold Properties

There are four categories of properties:

Table 12–1 Defined Property Names

Property Name 







Logging level 




Logging location 




Monitoring sample interval 




Decision history location 




Maximum number of CPUs for this processor set 




Minimum number of CPUs for this processor set 




CPUs pinned to this processor set 




Formatted string following poold's objective expression syntax




Formatted string following poold's expression syntax



Objective parameter 

User-assigned importance 

poold Features That Can Be Configured

You can configure these aspects of the daemon's behavior.

These options are specified in the pools configuration. You can also control the logging level from the command line by invoking poold.

poold Monitoring Interval

Use the property name system.poold.monitor-interval to specify a value in milliseconds.

poold Logging Information

Three categories of information are provided through logging. These categories are identified in the logs:

Use the property name system.poold.log-level to specify the logging parameter. If this property is not specified, the default logging level is NOTICE. The parameter levels are hierarchical. Setting a log level of DEBUG will cause poold to log all defined messages. The INFO level provides a useful balance of information for most administrators.

At the command line, you can use the poold command with the -l option and a parameter to specify the level of logging information generated.

The following parameters are available:

The parameter levels map directly onto their syslog equivalents. See Logging Location for more information about using syslog.

For more information about how to configure poold logging, see How to Set the poold Logging Level.

Configuration Information Logging

The following types of messages can be generated:


Problems accessing the libpool configuration, or some other fundamental, unanticipated failure of the libpool facility. Causes the daemon to exit and requires immediate administrative attention.


Problems due to unanticipated failures. Causes the daemon to exit and requires immediate administrative attention.


Problems with the user-specified parameters that control operation, such as unresolvable, conflicting utilization objectives for a resource set. Requires administrative intervention to correct the objectives. poold attempts to take remedial action by ignoring conflicting objectives, but some errors will cause the daemon to exit.


Warnings related to the setting of configuration parameters that, while technically correct, might not be suitable for the given execution environment. An example is marking all CPU resources as pinned, which means that poold cannot move CPU resources between processor sets.


Messages containing the detailed information that is needed when debugging configuration processing. This information is not generally used by administrators.

Monitoring Information Logging

The following types of messages can be generated:


Problems due to unanticipated monitoring failures. Causes the daemon to exit and requires immediate administrative attention.


Problems due to unanticipated monitoring error. Could require administrative intervention to correct.


Messages about resource control region transitions.


Messages about resource utilization statistics.


Messages containing the detailed information that is needed when debugging monitoring processing. This information is not generally used by administrators.

Optimization Information Logging

The following types of messages can be generated:


Messages could be displayed regarding problems making optimal decisions. Examples could include resource sets that are too narrowly constrained by their minimum and maximum values or by the number of pinned components.

Messages could be displayed about problems performing an optimal reallocation due to unforseen limitations. Examples could include removing the last processor from a processor set which contains a bound resource consumer.


Messages about usable configurations or configurations that will not be implemented due to overriding decision histories could be displayed.


Messages about alternate configurations considered could be displayed.


Messages containing the detailed information that is needed when debugging optimization processing. This information is not generally used by administrators.

Logging Location

The system.poold.log-location property is used to specify the location for poold logged output. You can specify a location of SYSLOG for poold output (see syslog(3C)).

If this property is not specified, the default location for poold logged output is /var/log/pool/poold.

When poold is invoked from the command line, this property is not used. Log entries are written to stderr on the invoking terminal.

Log Management With logadm

If poold is active, the logadm.conf file includes an entry to manage the default file /var/log/pool/poold. The entry is:

/var/log/pool/poold -N -s 512k

See the logadm(1M) and the logadm.conf(4) man pages.

How Dynamic Resource Allocation Works

This section explains the process and the factors that poold uses to dynamically allocate resources.

About Available Resources

Available resources are considered to be all of the resources that are available for use within the scope of the poold process. The scope of control is at most a single Solaris instance.

On a system that has zones enabled, the scope of an executing instance of poold is limited to the global zone.

Determining Available Resources

Resource pools encompass all of the system resources that are available for consumption by applications.

For a single executing Solaris instance, a resource of a single type, such as a CPU, must be allocated to a single partition. There can be one or more partitions for each type of resource. Each partition contains a unique set of resources.

For example, a machine with four CPUs and two processor sets can have the following setup:

pset 0: 0 1

pset 1: 2 3

where 0, 1, 2 and 3 after the colon represent CPU IDs. Note that the two processor sets account for all four CPUs.

The same machine cannot have the following setup:

pset 0: 0 1

pset 1: 1 2 3

It cannot have this setup because CPU 1 can appear in only one pset at a time.

Resources cannot be accessed from any partition other than the partition to which they belong.

To discover the available resources, poold interrogates the active pools configuration to find partitions. All resources within all partitions are summed to determine the total amount of available resources for each type of resource that is controlled.

This quantity of resources is the basic figure that poold uses in its operations. However, there are constraints upon this figure that limit the flexibility that poold has to make allocations. For information about available constraints, see Configuration Constraints.

Identifying a Resource Shortage

The control scope for poold is defined as the set of available resources for which poold has primary responsibility for effective partitioning and management. However, other mechanisms that are allowed to manipulate resources within this control scope can still affect a configuration. If a partition should move out of control while poold is active, poold tries to restore control through the judicious manipulation of available resources. If poold cannot locate additional resources within its scope, then the daemon logs information about the resource shortage.

Determining Resource Utilization

poold typically spends the greatest amount of time observing the usage of the resources within its scope of control. This monitoring is performed to verify that workload-dependent objectives are being met.

For example, for processor sets, all measurements are made across all of the processors in a set. The resource utilization shows the proportion of time that the resource is in use over the sample interval. Resource utilization is displayed as a percentage from 0 to 100.

Identifying Control Violations

The directives described in Configuration Constraints and Objectives are used to detect the approaching failure of a system to meet its objectives. These objectives are directly related to workload.

A partition that is not meeting user-configured objectives is a control violation. The two types of control violations are synchronous and asynchronous.

The following events cause asynchronous objective violations:

The contributions of objectives that are not related to workload are assumed to remain constant between evaluations of the objective function. Objectives that are not related to workload are only reassessed when a reevaluation is triggered through one of the asynchronous violations.

Determining Appropriate Remedial Action

When the resource controller determines that a resource consumer is short of resources, the initial response is that increasing the resources will improve performance.

Alternative configurations that meet the objectives specified in the configuration for the scope of control are examined and evaluated.

This process is refined over time as the results of shifting resources are monitored and each resource partition is evaluated for responsiveness. The decision history is consulted to eliminate reconfigurations that did not show improvements in attaining the objective function in the past. Other information, such as process names and quantities, are used to further evaluate the relevance of the historical data.

If the daemon cannot take corrective action, the condition is logged. For more information, see poold Logging Information.

Using poolstat to Monitor the Pools Facility and Resource Utilization

The poolstat utility is used to monitor resource utilization when pools are enabled on your system. This utility iteratively examines all of the active pools on a system and reports statistics based on the selected output mode. The poolstat statistics enable you to determine which resource partitions are heavily utilized. You can analyze these statistics to make decisions about resource reallocation when the system is under pressure for resources.

The poolstat utility includes options that can be used to examine specific pools and report resource set-specific statistics.

If zones are implemented on your system and you use poolstat in a non-global zone, information about the resources associated with the zone's pool is displayed.

For more information about the poolstat utility, see the poolstat(1M) man page. For poolstat task and usage information, see Using poolstat to Report Statistics for Pool-Related Resources.

poolstat Output

In default output format, poolstat outputs a heading line and then displays a line for each pool. A pool line begins with the pool ID and the name of the pool, followed by a column of statistical data for the processor set attached to the pool. Resource sets attached to more than one pool are listed multiple times, once for each pool.

The column headings are as follows:


Pool ID.


Pool name.


Resource set ID.


Resource set name.


Resource set type.


Minimum resource set size.


Maximum resource set size.


Current resource set size.


Measure of how much of the resource set is currently used.

This usage is calculated as the percentage of utilization of the resource set multiplied by the size of the resource set. If a resource set has been reconfigured during the last sampling interval, this value might be not reported. An unreported value appears as a hyphen (-).


Absolute representation of the load that is put on the resource set.

For more information about this property, see the libpool(3LIB) man page.

You can specify the following in poolstat output:

Tuning poolstat Operation Intervals

You can customize the operations performed by poolstat. You can set the sampling interval for the report and specify the number of times that statistics are repeated:


Tune the intervals for the periodic operations performed by poolstat. All intervals are specified in seconds.


Specify the number of times that the statistics are repeated. By default, poolstat reports statistics only once.

If interval and count are not specified, statistics are reported once. If interval is specified and count is not specified, then statistics are reported indefinitely.

Commands Used With the Resource Pools Facility

The commands described in the following table provide the primary administrative interface to the pools facility. For information on using these commands on a system that has zones enabled, see Resource Pools Used in Zones.

Man Page Reference 



Enables or disables the pools facility on your system. Activates a particular configuration or removes the current configuration and returns associated resources to their default status. If run without options, pooladm prints out the current dynamic pools configuration.


Enables the manual binding of projects, tasks, and processes to a resource pool. 


Provides configuration operations on pools and sets. Configurations created using this tool are instantiated on a target host by using pooladm.

If run with the info subcommand argument to the -c option, poolcfg displays information about the static configuration at /etc/pooladm.conf. If a file name argument is added, this command displays information about the static configuration held in the named file. For example, poolcfg -c info /tmp/newconfig displays information about the static configuration contained in the file /tmp/newconfig.


The pools system daemon. The daemon uses system targets and observable statistics to preserve the system performance goals specified by the administrator. If unable to take corrective action when goals are not being met, poold logs the condition.


Displays statistics for pool-related resources. Simplifies performance analysis and provides information that supports system administrators in resource partitioning and repartitioning tasks. Options are provided for examining specified pools and reporting resource set-specific statistics. 

A library API is provided by libpool (see the libpool(3LIB) man page). The library can be used by programs to manipulate pool configurations.