3 - C H A P T E R -

C H A P T E R 3

Compiling for OpenMP

This chapter describes how to compile programs that utilize the OpenMP API.

To run a parallelized program in a multithreaded environment, you must set the OMP_NUM_THREADS environment variable prior to program execution. This tells the runtime system the maximum number of threads the program can create. The default is 1. In general, set OMP_NUM_THREADS to the available number of processors on the target platform.

The compiler readme files contain information about limitations and known deficiencies regarding their OpenMP implementation. Readme files are viewable directly by invoking the compiler with the -xhelp=readme flag, or by pointing an HTML browser to the documentation index for the installed software at

file:/opt/SUNWspro/docs/index.html

3.1 Compiler Options To Use

To enable explicit parallelization with OpenMP directives, compile your program with the cc, CC, or f95 option flag -xopenmp. This flag can take an optional keyword argument. (The f95 compiler accepts both -xopenmp and -openmp as synonyms.)

The -xopenmp flag accepts the following keyword sub-options.

-xopenmp=parallel	Enables recognition of OpenMP pragmas. The minimum optimization level for `-xopenmp=parallel` is `-xO3`. The compiler changes the optimization from a lower level to `-xO3` if necessary, and issues a warning.
-xopenmp=noopt	Enables recognition of OpenMP pragmas. The compiler does not raise the level if it is lower than `-xO3`. If you explicitly set the optimization lower than `-xO3`, as in `-xO2` `-openmp=noopt` the compiler will issue an error. If you do not specify an optmization level with `-openmp=noopt`, the OpenMP pragmas are recognized, the program is parallelized accordingly, but no optimization is done. (This sub-option applies to `cc` and `f95` only; `CC` issues a warning if specified, and no OpenMP parallelization is done.)
-xopenmp=stubs	Disables recognition of OpenMP pragmas, links to stub library routines, and does not change the optimization level. Use this option if your application makes explicit calls to the OpenMP runtime library routines and you want to compile it to execute serially.
-xopenmp=none	Disables recognition of OpenMP pragmas and does not change the optimization level. (Default)

If you do not specify -xopenmp on the command line, the compiler assumes -xopenmp=none (disabling recognition of OpenMP pragmas).

If you specify -xopenmp but without a keyword sub-option, the compiler assumes -xopenmp=parallel.

Do not specify -xopenmp together with -xparallel or -xexplicitpar on the command line.

Specifying -xopenmp= with parallel , noopt , or stubs will define the _OPENMP preprocessor token to be YYYYMM (specifically 200203L for C/C++ and 200011 for Fortran 95).

When debugging OpenMP programs with dbx, compile with -xopenmp=noopt -g

The default optimization level for -xopenmp might change in future releases. Warning messages can be avoided by specifying an appropriate optimization level explicitly.

With Fortran 95, -xopenmp , -xopenmp=parallel, -xopenmp=noopt will add -stackvar automatically.

3.2 Fortran 95 OpenMP Validation

You can obtain a static, interprocedural validation of a Fortran 95 program's OpenMP directives by using the f95 compiler's global program checking feature. Enable OpenMP checking by compiling with the -XlistMP flag. (Diagnostic messages from -XlistMP appear in a separate file created with the name of the source file and a .lst extension). The compiler will diagnose the following violations and parallelization inhibitors:

Violations in the specifications of parallel directives, including improper nesting.

Parallelization inhibitors due to data usage, detected by interprocedural dependence analysis.

Parallelization inhibitors detected by interprocedural pointer analysis.

For example, compiling a source file ord.f with -XlistMP produces a diagnostic file ord.lst:

FILE  "ord.f"

     1  !$OMP PARALLEL

     2  !$OMP DO ORDERED

     3                  do i=1,100

     4                          call work(i)

     5                  end do

     6  !$OMP END DO

     7  !$OMP END PARALLEL

     9  !$OMP PARALLEL

    10  !$OMP DO

    11                  do i=1,100

    12                          call work(i)

    13                  end do

    14  !$OMP END DO

    15  !$OMP END PARALLEL

    16                  end

    17                  subroutine work(k)

    18  !$OMP ORDERED

**** ERR-OMP:  It is illegal for an ORDERED directive to bind to a

directive (ord.f, line 10, column 2) that does not have the

ORDERED clause specified.

    19                  write(*,*) k

    20  !$OMP END ORDERED

    21                  return

    22                  end

In this example, the ORDERED directive in subroutine WORK receives a diagnostic that refers to the second DO directive because it lacks an ORDERED clause.

3.3 OpenMP Environment Variables

The OpenMP specifications define four environment variables that control the execution of OpenMP programs. These are summarized in the following table. Additional multiprocessing environment variables affect execution of OpenMP programs and are not part of the OpenMP specifications. These are summarized in the following table.


Environment Variable	Function
OMP_SCHEDULE	Sets schedule type for DO, PARALLEL DO, parallel for, for, directives/pragmas with schedule type RUNTIME specified. If not defined, a default value of STATIC is used. value is "type[,chunk]" Example: `setenv OMP_SCHEDULE "GUIDED,4"`
OMP_NUM_THREADS or PARALLEL	Sets the number of threads to use during execution, unless set by a NUM_THREADS clause, or a call to OMP_SET_NUM_THREADS(). If not set, a default of 1 is used. value is a positive integer. (Current maximum is 128). For compatibility with legacy programs, setting the PARALLEL environment variable has the same effect as setting OMP_NUM_THREADS. However, if they are both set to different values, the runtime library will issue an error message. Example: `setenv OMP_NUM_THREADS 16`
OMP_DYNAMIC	Enables or disables dynamic adjustment of the number of threads available for execution of parallel regions. If not set, a default value of TRUE is used. value is either TRUE or FALSE. Example: `setenv OMP_DYNAMIC FALSE`
OMP_NESTED	Enables or disables nested parallelism. (Nested parallelism is not supported). value is either TRUE or FALSE. (This variable has no effect.) Example: `setenv OMP_NESTED FALSE`


Environment Variable	Function
SUNW_MP_WARN	Controls warning messages issued by the OpenMP runtime library. If set TRUE the runtime library issues warning messages to `stderr`; FALSE disables warning messages. The default is FALSE. Example: setenv SUNW_MP_WARN FALSE
SUNW_MP_THR_IDLE	Controls the end-of-task status of each helper thread executing the parallel part of a program. You can set the value to spin, sleep ns, or sleep nms. The default is SPIN -- the thread spins (or busy-waits) after completing a parallel task until a new parallel task arrives. Choosing SLEEP time specifies the amount of time a helper thread should spin-wait after completing a parallel task. If, while a thread is spinning, a new task arrives for the thread, the tread executes the new task immediately. Otherwise, the thread goes to sleep and is awakened when a new task arrives. time may be specified in seconds, `(`ns`)` or just `(`n`)`, or milliseconds, `(`nms`)`. SLEEP with no argument puts the thread to sleep immediately after completing a parallel task. SLEEP, SLEEP (0), SLEEP (0s), and SLEEP (0ms) are all equivalent. Example: setenv SUNW_MP_THR_IDLE SLEEP(50ms)
STACKSIZE	Sets the stack size for each thread. The value is in kilobytes. The default thread stack sizes are 4 Mb on 32-bit SPARC V8 platforms, and 8 Mb on 64-bit SPARC V9 platforms. Example: setenv STACKSIZE 8192 sets the thread stack size to 8 Mb

3.4 Stacks and Stack Sizes

The executing program maintains a main memory stack for the initial thread executing the program, as well as distinct stacks for each helper thread. Stacks are temporary memory address spaces used to hold arguments and automatic variables over subprogram or function references.

In general, the default main stack size is about 8 megabytes. Compiling Fortran programs with the f95 -stackvar option forces the allocation of local variables and arrays on the stack as if they were automatic variables. Use of -stackvar with OpenMP programs is implied with explicitly parallelized programs because it improves the optimizer's ability to parallelize calls in loops. (See the Fortran User's Guide for a discussion of the -stackvar flag.) However, this may lead to stack overflow if not enough memory is allocated for the stack.

Use the limit C-shell command, or the ulimit ksh/sh command, to display or set the size of the main stack.

Each helper thread of a multithreaded program has its own thread stack. This stack mimics the initial (or main) thread stack but is unique to the thread. The thread's PRIVATE arrays and variables (local to the thread) are allocated on the thread stack. The default size is 4 megabytes on 32-bit systems and 8 megabytes on 64-bit systems. The size of the helper thread stack is set with the STACKSIZE environment variable.

demo% setenv STACKSIZE 16384   <-Set thread stack size to 16 Mb (C shell)

demo% STACKSIZE=16384          <-Same, using Bourne/Korn shell

demo% export STACKSIZE

Finding the best stack size might have to be determined by trial and error. If the stack size is too small for a thread to run it may cause silent data corruption in neighboring threads, or segmentation faults. If you are unsure about stack overflows, compile your Fortran or C programs with the -xcheck=stkovf flag to force a segmentation fault on stack overflow. This stops the program before any data corruption can occur.