Solaris Containers: Resource Management and Solaris Zones Developer's Guide

Chapter 6 Resource Pools

This chapter describes resource pools and their properties.

Overview of Resource Pools

Resource pools provide a framework for managing processor sets and thread scheduling classes. Resource pools are used for partitioning machine resources. Resource pools enable you to separate workloads so that workload consumption of certain resources does not overlap. The resource reservation helps to achieve predictable performance on systems with mixed workloads.

For an overview of resource pools and example commands for administering resource pools, see Chapter 12, Resource Pools (Overview), in System Administration Guide: Solaris Containers-Resource Management and Solaris Zones and Chapter 13, Creating and Administering Resource Pools (Tasks), in System Administration Guide: Solaris Containers-Resource Management and Solaris Zones.

A processor set groups the CPUs on a system into a bounded entity, on which a process or processes can run exclusively. Processes cannot extend beyond the processor set, nor can other processes extend into the processor set. A processor set enables tasks of similar characteristics to be grouped together and a hard upper boundary for CPU use to be set.

The resource pool framework allows the definition of a soft processor set with a maximum and minimum CPU count requirement. Additionally, the framework provides a hard-defined scheduling class for that processor set.

A zone can be bound to a resource pool through the pool property of the zone configuration. The zone is bound to the specified pool upon creation of the zone. The pool configuration can be changed only from the global zone. Zones cannot span multiple pools. All processes in a zone run in the same pool. However, multiple zones can bind to the same resource pool.

A resource pool defines:

Processor set groups
Scheduling class

Scheduling Class

Scheduling classes provide different CPU access characteristics to threads that are based on algorithmic logic. The scheduling classes include:

Realtime scheduling class
Interactive scheduling class
Fixed priority scheduling class
Timesharing scheduling class
Fair share scheduling class

For an overview of fair share scheduler and example commands for administering the fair share scheduler, see Chapter 8, Fair Share Scheduler (Overview), in System Administration Guide: Solaris Containers-Resource Management and Solaris Zones and Chapter 9, Administering the Fair Share Scheduler (Tasks), in System Administration Guide: Solaris Containers-Resource Management and Solaris Zones.

Do not mix scheduling classes in a set of CPUs. If scheduling classes are mixed in a CPU set, system performance might become erratic and unpredictable. Use processor sets to segregate applications by their characteristics. Assign scheduling classes under which the application best performs. For more information about the characteristics of an individual scheduling class, see priocntl(1).

For an overview of resource pools and a discussion of when to use pools, see Chapter 6, Resource Pools.

Dynamic Resource Pool Constraints and Objectives

The libpool library defines properties that are available to the various entities that are managed within the pools facility. Each property falls into the following categories:

Configuration constraints

A constraint defines boundaries of a property. Typical constraints are the maximum and minimum allocations specified in the libpool configuration.

Objective

An objective changes the resource assignments of the current configuration to generate new candidate configurations that observe the established constraints. An objective has the following categories:

Workload dependent: A workload-dependent objective varies according to the conditions imposed by the workload. An example of the workload dependent objective is the utilization objective.
Workload independent: A workload-independent objective does not vary according to the conditions imposed by the workload. An example of the workload independent objective is the cpu locality objective.

An objective can take an optional prefix to indicate the importance of the objective. The objective is multiplied by this prefix, which is an integer from 0 to INT64_MAX,, to determine the significance of the objective.

For usage examples, see How to Set Configuration Constraints in System Administration Guide: Solaris Containers-Resource Management and Solaris Zones and How to Define Configuration Objectives in System Administration Guide: Solaris Containers-Resource Management and Solaris Zones.

System Properties

system.bind-default (writable boolean): If the specified pool is not found in <filename>/etc/project</filename>, bind to pool with the pool.default property set to TRUE.
system.comment (writable string): User description of system. system.comment is not used by the default pools commands, except when a configuration is initiated by the poolcfg utility. In this case, the system puts an informative message in the system.comment property for that configuration.
system.name (writable string): User name for the configuration.
system.version (read-only integer): libpool version required to manipulate this configuration.

Pools Properties

All pools properties except pool.default and pool.sys_id are writable.

pool.active (writable boolean)

If TRUE, mark this pool as active.

pool.comment (writable string)

User description of pool.

pool.default (read-only boolean)

If TRUE, mark this pool as the default pool. See the system.bind-default property.

pool.importance (writable integer)

Relative importance of this pool. Used for possible resource dispute resolution.

pool.name (writable string)

User name for pool. setproject(3PROJECT) uses pool.name as the value for the project.pool project attribute in the project(4) database.

pool.scheduler (writable string)

Scheduler class to which consumers of this pool are bound. This property is optional and if not specified, the scheduler bindings for consumers of this pool are not affected. For more information about the characteristics of an individual scheduling class, see priocntl(1). Scheduler classes include:

RT for realtime scheduler
TS for timesharing scheduler
IA for interactive scheduler
FSS for fair share scheduler
FX for fixed priority scheduler

pool.sys_id (read-only integer)

This is the system-assigned pool ID.

Processor Set Properties

pset.comment (writable string): User description of resource.
pset.default (read-only boolean): Identifies the default processor set.
pset.escapable (writable boolean): Represents whether PSET_NOESCAPE is set for this pset. See the pset_setattr(2) man page.
pset.load (read-only unsigned integer): The load for this processor set. The lowest value is 0. The value increases in a linear fashion with the load on the set, as measured by the number of jobs in the system run queue.
pset.max (writable unsigned integer): Maximum number of CPUs that are permitted in this processor set.
pset.min (writable unsigned integer): Minimum number of CPUs that are permitted in this processor set.
pset.name (writable string): User name for the resource.
pset.size (read-only unsigned integer): Current number of CPUs in this processor set.
pset.sys_id (read-only integer): System-assigned processor set ID.
pset.type (read-only string): Names the resource type. Value for all processor sets is pset.
pset.units (read-only string): Identifies the meaning of size-related properties. The value for all processor sets is population.
cpu.comment (writable string): User description of CPU

Using `libpool` to Manipulate Pool Configurations

The libpool(3LIB) pool configuration library defines the interface for reading and writing pools configuration files. The library also defines the interface for committing an existing configuration to becoming the running operating system configuration. The <pool.h> header provides type and function declarations for all library services.

The resource pools facility brings together process-bindable resources into a common abstraction that is called a pool. Processor sets and other entities can be configured, grouped, and labelled in a persistent fashion. Workload components can be associated with a subset of a system's total resources. The libpool(3LIB) library provides a C language API for accessing the resource pools facility. The pooladm(1M), poolbind(1M), and poolcfg(1M) make the resource pools facility available through command invocations from a shell.

Manipulate `psets`

The following list contains the functions associated with creating or destroying psets and manipulating psets.

processor_bind(2): Bind an LWP (lightweight process) or set of LWPs to a specified processor.
pset_assign(2): Assign a processor to a processor set.
pset_bind(2): Bind one or more LWPs (lightweight processes) to a processor set.
pset_create(2): Create an empty processor set that contains no processors.
pset_destroy(2): Destroy a processor set and release the associated constituent processors and processes.
pset_setattr(2), pset_getattr(2): Set or get processor set attributes.

Resource Pools API Functions

This section lists all of the resource pool functions. Each function has a link to the man page and a short description of the function's purpose. The functions are divided into two groups, depending on whether the function performs an action or a query:

The imported interfaces for libpool for swap sets is identical to the ones defined in this document.

Functions for Operating on Resource Pools and Associated Elements

The interfaces listed in this section are for performing actions related to pools and the associated elements.

pool_associate(3POOL): Associate a resource with a specified pool.
pool_component_to_elem(3POOL): Convert specified component to the pool element type.
pool_conf_alloc(3POOL): Create a pool configuration.
pool_conf_close(3POOL): Close the specified pool configuration and release the associated resources.
pool_conf_commit(3POOL): Commit changes made to the specified pool configuration to permanent storage.
pool_conf_export(3POOL): Save the given configuration to the specified location.
pool_conf_free(3POOL): Release a pool configuration.
pool_conf_open(3POOL): Create a pool configuration at the specified location.
pool_conf_remove(3POOL): Removes the permanent storage for the configuration.
pool_conf_rollback(3POOL): Restore the configuration state to the state that is held in the pool configuration's permanent storage.
pool_conf_to_elem(3POOL): Convert specified pool configuration to the pool element type.
pool_conf_update(3POOL): Update the library snapshot of kernel state.
pool_create(3POOL): Create a new pool with the default properties and with default resources for each type.
pool_destroy(3POOL): Destroy the specified pool. The associated resources are not modified.
pool_dissociate(3POOL): Remove the association between the given resource and pool.
pool_put_property(3POOL): Set the named property on the element to the specified value.
pool_resource_create(3POOL): Create a new resource with the specified name and type for the provided configuration.
pool_resource_destroy(3POOL): Remove the specified resource from the configuration file.
pool_resource_to_elem(3POOL): Convert specified pool resource to the pool element type.
pool_resource_transfer(3POOL): Transfer basic units from the source resource to the target resource.
pool_resource_xtransfer(3POOL): Transfer the specified components from the source resource to the target resource.
pool_rm_property(3POOL): Remove the named property from the element.
pool_set_binding(3POOL): Bind the specified processes to the resources that are associated with pool on the running system.
pool_set_status(3POOL): Modify the current state of the pools facility.
pool_to_elem(3POOL): Convert specified pool to the pool element type.
pool_value_alloc(3POOL): Allocate and return an opaque container for a pool property value.
pool_value_free(3POOL): Release an allocated property values.
pool_value_set_bool(3POOL): Set a property value of type boolean.
pool_value_set_double(3POOL): Set a property value of type double.
pool_value_set_int64(3POOL): Set a property value of type int64.
pool_value_set_name(3POOL): Set a name=value pair for a pool property.
pool_value_set_string(3POOL): Copy the string that was passed in.
pool_value_set_uint64(3POOL): Set a property value of type uint64.

Functions for Querying Resource Pools and Associated Elements

The interfaces listed in this section are for performing queries related to pools and the associated elements.

pool_component_info(3POOL): Return a string that describes the given component.
pool_conf_info(3POOL): Return a string describing the entire configuration.
pool_conf_location(3POOL): Return the location string that was provided to pool_conf_open() for the given specified configuration.
pool_conf_status(3POOL): Return the validity status for a pool configuration.
pool_conf_validate(3POOL): Check the validity of the contents of the given configuration.
pool_dynamic_location(3POOL): Return the location that was used by the pools framework to store the dynamic configuration.
pool_error(3POOL): Return the error value of the last failure that was recorded by calling a resource pool configuration library function.
pool_get_binding(3POOL): Return the name of the pool on the running system that contains the set of resources to which the specified process is bound.
pool_get_owning_resource(3POOL): Return the resource that currently contains the specified component.
pool_get_pool(3POOL): Return the pool with the specified name from the provided configuration.
pool_get_property(3POOL): Retrieve the value of the named property from the element.
pool_get_resource(3POOL): Return the resource with the given name and type from the provided configuration.
pool_get_resource_binding(3POOL): Return the name of the pool on the running system that contains the set of resources to which the given process is bound.
pool_get_status(3POOL): Retrieve the current state of the pools facility.
pool_info(3POOL): Return a description of the specified pool.
pool_query_components(3POOL): Retrieve all resource components that match the specified list of properties.
pool_query_pool_resources(3POOL): Return a null-terminated array of resources currently associated with the pool.
pool_query_pools(3POOL): Return the list of pools that match the specified list of properties.
pool_query_resource_components(3POOL): Return a null-terminated array of the components that make up the specified resource.
pool_query_resources(3POOL): Return the list of resources that match the specified list of properties.
pool_resource_info(3POOL): Return a description of the specified resource.
pool_resource_type_list(3POOL): Enumerate the resource types that are supported by the pools framework on this platform.
pool_static_location(3POOL): Return the location that was used by the pools framework to store the default configuration for pools framework instantiation.
pool_strerror(3POOL): Return a description of each valid pool error code.
pool_value_get_bool(3POOL): Get a property value of type boolean.
pool_value_get_double(3POOL): Get a property value of type double.
pool_value_get_int64(3POOL): Get a property value of type int64.
pool_value_get_name(3POOL): Return the name that was assigned to the specified pool property.
pool_value_get_string(3POOL): Get a property value of type string.
pool_value_get_type(3POOL): Return the type of the data that is contained by the specified pool value.
pool_value_get_uint64(3POOL): Get a property value of type uint64.
pool_version(3POOL): Get the version number of the pool library.
pool_walk_components(3POOL): Invoke callback on all components that are contained in the resource.
pool_walk_pools(3POOL): Invoke callback on all pools that are defined in the configuration.
pool_walk_properties(3POOL): Invoke callback on all properties defined for the given element.
pool_walk_resources(3POOL): Invoke callback on all resources that are associated with the pool.

Resource Pool Code Examples

This section contains code examples of the resource pools interface.

Ascertain the Number of CPUs in the Resource Pool

sysconf(3C) provides information about the number of CPUs on an entire system. The following example provides the granularity of ascertaining the number of CPUs that are defined in a particular application's pools pset.

The key points for this example include the following:

pvals[] should be a NULL terminated array.
pool_query_pool_resources() returns a list of all resources that match the pvals array type pset from the application's pool my_pool. Because a pool can have only one instance of the pset resource, each instance is always returned in nelem. reslist[] contains only one element, the pset resource.

pool_value_t *pvals[2] = {NULL};  /* pvals[] should be NULL terminated */

/* NOTE: Return value checking/error processing omitted */
/* in all examples for brevity */

conf_loc = pool_dynamic_location();
conf = pool_conf_alloc();
pool_conf_open(conf, conf_loc, PO_RDONLY);
my_pool_name = pool_get_binding(getpid());
my_pool = pool_get_pool(conf, my_pool_name);
pvals[0] = pool_value_alloc();
pvals2[2] = { NULL, NULL };
pool_value_set_name(pvals[0], "type");
pool_value_set_string(pvals[0], "pset");

reslist = pool_query_pool_resources(conf, my_pool, &nelem, pvals);
pool_value_free(pvals[0]);
pool_query_resource_components(conf, reslist[0], &nelem, NULL);
printf("pool %s: %u cpu", my_pool_ name, nelem);
pool_conf_close(conf);

List All Resource Pools

The following example lists all resource pools defined in an application's pools pset.

The key points of the example include the following:

Open the dynamic conf file read-only, PO_RDONLY. pool_query_pools() returns the list of pools in pl and the number of pools in nelem. For each pool, call pool_get_property() to get the pool.name property from the element into the pval value.
pool_get_property() calls pool_to_elem() to convert the libpool entity to an opaque value. pool_value_get_string() gets the string from the opaque pool value.

conf	= pool_conf_alloc();
pool_conf_open(conf, pool_dynamic_location(), PO_RDONLY);
pl = pool_query_pools(conf, &nelem, NULL);
pval = pool_value_alloc();
for (i = 0; i < nelem; i++) {
    pool_get_property(conf, pool_to_elem(conf, pl[i]), "pool.name", pval);
    pool_value_get_string(pval, &fname);
    printf("%s\n", name);
}
pool_value_free(pval);
free(pl);
pool_conf_close(conf);

Report Pool Statistics for a Given Pool

The following example reports statistics for the designated pool.

The key points for the example include the following:

pool_query_pool_resources() gets a list of all resources in rl. Because the last argument to pool_query_pool_resources() is NULL, all resources are returned. For each resource, the name, load and size properties are read, and printed.
The call to strdup() allocates local memory and copies the string returned by get_string(). The call to get_string() returns a pointer that is freed by the next call to get_property(). If the call to strdup() is not included, subsequent references to the string(s) could cause the application to fail with a segmentation fault.

printf("pool %s\n:" pool_name);
pool = pool_get_pool(conf, pool_name);
rl = pool_query_pool_resources(conf, pool, &nelem, NULL);
for (i = 0; i < nelem; i++) {
  pool_get_property(conf, pool_resource_to_elem(conf, rl[i]), "type", pval);
  pool_value_get_string(pval, &type);
  type = strdup(type);
  snprintf(prop_name, 32, "%s.%s", type, "name");
  pool_get_property(conf, pool_resource_to_elem(conf, rl[i]), 
        prop_name, pval);
  pool_value_get_string(val, &res_name);
  res_name = strdup(res_name);
  snprintf(prop_name, 32, "%s.%s", type, "load");
  pool_get_property(conf, pool_resource_to_elem(conf, rl[i]), 
        prop_name, pval);
  pool_value_get_uint64(val, &load);
  snprintf(prop_name, 32, "%s.%s", type, "size");
  pool_get_property(conf, pool_resource_to_elem(conf, rl[i]), 
        prop_name, pval);
  pool_value_get_uint64(val, &size);
  printf("resource %s: size %llu load %llu\n", res_name, size, load);
  free(type);
  free(res_name);
}
free(rl);

Set `pool.comment` Property and Add New Property

The following example sets the pool.comment property for the pset. The example also creates a new property in pool.newprop.

The key point for the example includes the following:

In the call to pool_conf_open(), using PO_RDWR on a static configuration file, requires the caller to be root.
To commit these changes to the pset after running this utility, issue a pooladm -c command. To have the utility commit the changes, call pool_conf_commit() with a nonzero second argument.

pool_set_comment(const char *pool_name, const char *comment)
{
  pool_t *pool;
  pool_elem_t *pool_elem;
  pool_value_t *pval = pool_value_alloc(); 
  pool_conf_t  *conf = pool_conf_alloc();
  /* NOTE: need to be root to use PO_RDWR on static configuration file */
  pool_conf_open(conf, pool_static_location(), PO_RDWR);
  pool = pool_get_pool(conf,  pool_name);
  pool_value_set_string(pval, comment);
  pool_elem = pool_to_elem(conf, pool);
  pool_put_property(conf, pool_elem, "pool.comment", pval);
  printf("pool %s: pool.comment set to %s\n:" pool_name, comment);
  /* Now, create a new property, customized to installation site */
  pool_value_set_string(pval, "New String Property");
  pool_put_property(conf, pool_elem, "pool.newprop", pval);
  pool_conf_commit(conf, 0); /* NOTE: use 0 to ensure only */
                             /* static file gets updated */
  pool_value_free(pval);
  pool_conf_close(conf);
  pool_conf_free(conf);
  /* NOTE: Use "pooladm -c" later, or pool_conf_commit(conf, 1) */
  /* above for changes to the running system */
}

An alternative way of modifying a pool's comment and adding a new pool property is to use poolcfg(1M).

poolcfg -c 'modify pool pool-name (string pool.comment = "cmt-string")'
poolcfg -c 'modify pool pool-name (string pool.newprop = 
                                   "New String Property")'

Programming Issues Associated With Resource Pools

Consider the following issues when writing your application.

Each site can add its own list of properties to the pools configuration.

Multiple configurations can be maintained in multiple configuration files. The system administrator can commit different files to reflect changes to the resource consumption at different time slots. These time slots can include different times of the day, week, month, or seasons depending on load conditions.
Resource sets can be shared between pools, but a pool has only one resource set of a given type. So, the pset_default can be shared between the default and a particular application's database pools.
Use pool_value_*() interfaces carefully. Keep in mind the memory allocation issues for string pool values. See Report Pool Statistics for a Given Pool.