Locality Groups Overview

Language:

Shared memory multiprocessor systems contain multiple CPUs. Each CPU can access all of the memory in the system. In some shared memory multiprocessors, the memory architecture enables each CPU to access some areas of memory more quickly than other areas.

When a system with such a memory architecture runs the Oracle Solaris software, providing information to the kernel about the shortest access time between a given CPU and a given area of memory can improve the system performance. The locality group (lgroup) abstraction handles this information. The lgroup abstraction is part of the Memory Placement Optimization (MPO) feature. For more information, see Memory and Thread Placement Optimization Developer’s Guide.

An lgroup is a set of CPU–like and memory–like device in which each CPU in the set can access any memory in that set within a bounded latency interval. The value of the latency interval represents the least common latency between all the CPUs and all the memory in that lgroup. The latency bound that defines an lgroup does not restrict the maximum latency between members of that lgroup.

Lgroups are hierarchical. The lgroup hierarchy is a Directed Acyclic Graph (DAG) and is similar to a tree, except that an lgroup might have more than one parent. The root lgroup contains all the resources in the system and can include child lgroups. The root lgroup have the highest latency value of all the lgroups in the system. All of its child lgroups have lower latency values. The lgroups closer to the root have a higher latency while lgroups closer to leaves have lower latency.

A system in which all the CPUs can access all the memory with same latency can be represented with a single lgroup (see Single Locality Group Schematic). A system in which some of the CPUs can access some areas of memory in a shorter time than other areas can be represented by using multiple lgroups (see Multiple Locality Groups Schematic).

Figure 3 Single Locality Group Schematic

image:Image illustrates single locality group schematic. All CPUs in the system can access the memory in a comparable time frame.

Figure 4 Multiple Locality Groups Schematic

image:Image illustrates multiple locality groups schematic. The CPU and memory resources are grouped by bounded latency intervals.

The lgroup hierarchy helps to find the nearest resources in the system. Each thread is assigned a home lgroup upon creation. The operating system allocates resources for the thread from the thread's home lgroup by default. For example, the Oracle Solaris kernel schedules a thread to run on the CPUs in the thread's home lgroup and allocates the thread's memory in the thread's home lgroup by default. If the desired resources are not available from the thread's home lgroup, the kernel can traverse the lgroup hierarchy to find the nearest resources from the home lgroup's parents. If the desired resources are not available in the home lgroup's parents, the kernel continues to traverse the lgroup hierarchy to the next ancestor lgroups of the home lgroup. The root lgroup is the ancestor of all other lgroups in a system and contains all the resources of the system.

The lgroup APIs export the lgroup abstraction for applications to use for observability and performance tuning. A library, called liblgrp, contains the lgroup APIs.

Applications can use the APIs to perform the following tasks:

Traverse the group hierarchy
Discover the contents and characteristics of a given lgroup
Affect the thread and memory placement on lgroups