test

Sun Studio 12 Update 1: OpenMP API User's Guide

4.2 Control of Nested Parallelism

Nested parallelism can be controlled at runtime by setting various environment variables prior to execution of the program.

4.2.1 OMP_NESTED

Nested parallelism can be enabled or disabled by setting the OMP_NESTED environment variable or calling omp_set_nested().

The following example has three levels of nested parallel constructs.

Example 4–1 Nested Parallelism Example


#include <omp.h>
#include <stdio.h>
void report_num_threads(int level)
{
    #pragma omp single
    {
        printf("Level %d: number of threads in the team - %d\n",
                  level, omp_get_num_threads());
    }
 }
int main()
{
    omp_set_dynamic(0);
    #pragma omp parallel num_threads(2)
    {
        report_num_threads(1);
        #pragma omp parallel num_threads(2)
        {
            report_num_threads(2);
            #pragma omp parallel num_threads(2)
            {
                report_num_threads(3);
            }
        }
    }
    return(0);
}

Compiling and running this program with nested parallelism enabled produces the following (sorted) output:


% setenv OMP_NESTED TRUE
% a.out
Level 1: number of threads in the team - 2
Level 2: number of threads in the team - 2
Level 2: number of threads in the team - 2
Level 3: number of threads in the team - 2
Level 3: number of threads in the team - 2
Level 3: number of threads in the team - 2
Level 3: number of threads in the team - 2

Compare with running the same program but with nested parallelism disabled:


% setenv OMP_NESTED FALSE
% a.out
Level 1: number of threads in the team - 2
Level 2: number of threads in the team - 1
Level 3: number of threads in the team - 1
Level 2: number of threads in the team - 1
Level 3: number of threads in the team - 1

4.2.2 OMP_THREAD_LIMIT

The OMP_THREAD_LIMIT environment variable sets the maximum number of OpenMP threads to use for the whole OpenMP program. The defaut number in Sun's implementation is 1024. If this environment variable is set to one, then all parallel regions will be executed by one thread.

The following example shows that a parallel region can get fewer threads if OMP_THREAD_LIMIT is set. The code is the same as above. The number of threads needed for all the parallel regions to be active at the same time is 8. If we set OMP_THREAD_LIMIT to 6, two of the four inner-most parallel regions may not be able to get all the helper threads they ask for. One possible result is shown below.


% setenv OMP_NESTED TRUE
% setenv OMP_THREAD_LIMIT 6
% a.out
Level 1: number of threads in the team - 2
Level 2: number of threads in the team - 2
Level 2: number of threads in the team - 2
Level 3: number of threads in the team - 2
Level 3: number of threads in the team - 2
Level 3: number of threads in the team - 1
Level 3: number of threads in the team - 1

4.2.3 OMP_MAX_ACTIVE_LEVELS

The environment variable OMP_MAX_ACTIVE_LEVELS controls the maximum depth of nested active parallel regions that require more than one thread.

Any active parallel region that has an active nested depth greater than the value of this environment variable will be executed by only one thread. A parallel region is considered active if it it has no IF clause, or if it has an IF clause that evaluates to true. The default maximum number of active nesting levels is 4.

The following code will create 4 levels of nested parallel regions. If OMP_MAX_ACTIVE_LEVELS is set to 2, then nested parallel regions at nested depth of 3 and 4 are executed single-threaded.


#include <omp.h>
#include <stdio.h>
#define DEPTH 5
void report_num_threads(int level)
{
    #pragma omp single
    {
        printf("Level %d: number of threads in the team - %d\n",
               level, omp_get_num_threads());
    }
}
void nested(int depth)
{
    if (depth == DEPTH)
        return;

    #pragma omp parallel num_threads(2)
    {
        report_num_threads(depth);
        nested(depth+1);
    }
}
int main()
{
    omp_set_dynamic(0);
    omp_set_nested(1);
    nested(1);
    return(0);
}

Compiling and running this program with a maximum nesting level of 4 gives the following possible output. (Actual results will depend on how the OS schedules threads.)


% setenv OMP_MAX_ACTIVE_LEVELS 4
% a.out |sort 
Level 1: number of threads in the team - 2
Level 2: number of threads in the team - 2
Level 2: number of threads in the team - 2
Level 3: number of threads in the team - 2
Level 3: number of threads in the team - 2
Level 3: number of threads in the team - 2
Level 3: number of threads in the team - 2
Level 4: number of threads in the team - 2
Level 4: number of threads in the team - 2
Level 4: number of threads in the team - 2
Level 4: number of threads in the team - 2
Level 4: number of threads in the team - 2
Level 4: number of threads in the team - 2
Level 4: number of threads in the team - 2
Level 4: number of threads in the team - 2

Running with the nesting level set at 2 gives the following as a possible result:


% setenv OMP_MAX_ACTIVE_LEVELS 2
% a.out |sort 
Level 1: number of threads in the team - 2
Level 2: number of threads in the team - 2
Level 2: number of threads in the team - 2
Level 3: number of threads in the team - 1
Level 3: number of threads in the team - 1
Level 3: number of threads in the team - 1
Level 3: number of threads in the team - 1
Level 4: number of threads in the team - 1
Level 4: number of threads in the team - 1
Level 4: number of threads in the team - 1
Level 4: number of threads in the team - 1

Again, these examples only show some possible results. Actual results will depend on how the OS schedules threads.