2.3 Processor Binding

With processor binding, the programmer instructs the operating system that a thread in the program should run on the same processor throughout the execution of the program.

Processor binding when used with static scheduling benefits applications that exhibit a certain data reuse pattern where data accessed by a thread in a parallel or worksharing region will be in the local cache from a previous invocation of a parallel or worksharing region.

From the hardware point of view, a computer system is composed of one or more physical processors. From the operating system point of view, each of these physical processors maps to one or more virtual processors onto which threads in a program can be run. If n virtual processors are available, then n threads can be scheduled to run at the same time. Depending on the system, a virtual processor may be a processor, a core, a hardware thread, and so on.

For example, the SPARC T3 physical processor has eight cores, and each core can run eight simultaneous processing threads. From the Oracle Solaris system point of view, there are 64 virtual processors onto which threads can be scheduled to run. On Oracle Solaris platforms, the number of virtual processors can be determined by using the psrinfo(1M) command. On Linux systems, the text file /proc/cpuinfo provides information about available processors.

When the operating system binds threads to processors, they are in effect bound to specific virtual processors, not physical processors.

Set the SUNW_MP_PROCBIND environment variable to bind threads in an OpenMP program. The value specified for SUNW_MP_PROCBIND can be one of the following:

• The string "TRUE" or "FALSE" or "COMPACT" or "SCATTER" (or lowercase "true" or "false" or "compact" or "scatter"). For example, % setenv SUNW_MP_PROCBIND "TRUE"

• A non-negative integer. For example, % setenv SUNW_MP_PROCBIND "2"

• A list of two or more non-negative integers separated by one or more spaces. For example, % setenv SUNW_MP_PROCBIND "2 2 4 6" will bind two threads to processor 2, one will be bound to processor 4, and one will be bound to processor 6, if four threads are used.

• Two non-negative integers, n1 and n2, separated by a minus sign ("-"). n1 must be less than or equal to n2. For example, % setenv SUNW_MP_PROCBIND "0-6"

Note that the non-negative integers referred to above denote logical identifiers (IDs). Logical IDs may be different from virtual processor IDs. The difference will be explained in the next section.

2.3.1 Virtual and Logical Processor IDs

Each virtual processor in a system has a unique processor ID. The Oracle Solaris psrinfo(1M) command displays information about the virtual processors in a system, including their virtual processor IDs. The prtdiag(1M) command displays system configuration and diagnostic information.

You can use psrinfo -pv to list all physical processors in the system and the virtual processors that are associated with each physical processor.

Virtual processor IDs could be sequential, but gaps in the IDs can also occur. For example, on a Sun Fire 4810 with 8 UltraSPARC IV processors (16 cores), the virtual processor IDs might be: 0, 1, 2, 3, 8, 9, 10, 11, 512, 513, 514, 515, 520, 521, 522, 523.

Logical processor IDs on the other hand are consecutive integers that start with 0. If the number of virtual processors available in the system is n, then their logical IDs are 0, 1, ..., n-1, in the order presented by psrinfo(1M).

For an interpretation of the values accepted by SUNW_MP_PROCBIND, see 2.3.2 Interpreting the Value Specified for SUNW_MP_PROCBIND.

2.3.2 Interpreting the Value Specified for SUNW_MP_PROCBIND

If the value specified for SUNW_MP_PROCBIND is FALSE, the threads will not be bound to any processors. This is the default setting.

If the value specified for SUNW_MP_PROCBIND is TRUE, then the threads will be bound to virtual processors in a round-robin fashion. The starting processor for the binding is determined by the runtime library with the goal of achieving best performance.

If the value specified for SUNW_MP_PROCBIND is COMPACT, then the threads will be bound to virtual processors that are as close together as possible on the system. COMPACT allows threads to share data caches and thus improve data locality.

If the value specified for SUNW_MP_PROCBIND is SCATTER, then the threads will be bound to virtual processors that are far apart. This will allow higher memory bandwidth for each of the threads. SCATTER is the opposite of COMPACT.

If the value specified for SUNW_MP_PROCBIND is a non-negative integer, then that integer denotes the starting logical ID of the virtual processor to which threads should be bound. Threads will be bound to virtual processors in a round-robin fashion starting with the processor with the specified logical ID and wrapping around to the processor with logical ID 0 after binding to the processor with logical ID n-1.

If the value specified for SUNW_MP_PROCBIND is a list of two or more non-negative integers, then threads will be bound in a round-robin fashion to virtual processors with the specified logical IDs. Processors with logical IDs other than those specified will not be used.

If the value specified for SUNW_MP_PROCBIND is two non-negative integers separated by a minus ("-"), then threads will be bound in a round-robin fashion to virtual processors in the range that begins with the first logical ID and ends with the second logical ID. Processors with logical IDs other than those included in the range will not be used.

If the value specified for SUNW_MP_PROCBIND does not conform to one of the forms described above, or if an invalid logical ID is given, then an error message will be emitted and execution of the program will terminate.

If the number of threads in an OpenMP program is is greater than the number of virtual processors available, some virtual processors will have more than one thread bound to them. This may negatively impact performance.

2.3.3 Interaction With OS Processor Sets

A processor set can be specified using the psrset utility on Oracle Solaris platforms, or the taskset command on Linux platforms. SUNW_MP_PROCBIND does not take processor sets into account. If you use processor sets, then it is your responsibility to ensure that the setting of SUNW_MP_PROCBIND is consistent with the processor set used. Otherwise, the setting of SUNW_MP_PROCBIND will override the processor set setting on Linux systems, while on Oracle Solaris systems an error message will be issued.