Multithreaded Programming

When you are linked to one of the thread-safe libraries, Sun MPI calls are thread safe, in accordance with basic tenets of thread safety for MPI mentioned in the MPI-2 specification [Document for a Standard Message-Passing Interface. Please see the preface of this document for more information about this and other recommended reference material. ] . This means that:

• When two concurrently running threads make MPI calls, the outcome will be as if the calls executed in some order.

• Blocking MPI calls will block the calling thread only. A blocked calling thread will not prevent progress of other runnable threads on the same process, nor will it prevent them from executing MPI calls. Thus, multiple sends and receives are concurrent.

Guidelines for Thread-Safe Programming

Each thread within an MPI process may issue MPI calls; however, threads are not separately addressable. That is, the rank of a send or receive call identifies a process, not a thread, meaning that no order is defined for the case where two threads call MPI_Recv with the same tag and communicator. Such threads are said to be in conflict.

If threads within the same application post conflicting communication calls, data races will result. You can prevent such data races by using distinct communicators or tags for each thread.

In general, you will need to adhere to these guidelines:

• You must not have a request serviced by more than one thread. Although you may have an operation posted in one thread and then completed in another, you may not have the operation completed in more than one thread.

• A data type or communicator must not be freed by one thread while it is in use by another thread.

• Once MPI_Finalize has been called, subsequent calls in any thread will fail.

• You must ensure that a sufficient number of lightweight processes (LWPs) are available for your multithreaded program. Failure to do so may degrade performance or even result in deadlock.

• You cannot stub the thread calls in your multithreaded program by omitting the threads libraries in the link line. The libmpi.so library automatically calls in the threads libraries, which effectively overrides any stubs.

The following sections describe more specific guidelines that apply for some routines. They also include some general considerations for collective calls and communicator operations that you should be aware of.

MPI_Wait, MPI_Waitall, MPI_Waitany, MPI_Waitsome

In a program where two or more threads call one of these routines, you must ensure that they are not waiting for the same request. Similarly, the same request cannot appear in the array of requests of multiple concurrent wait calls.

MPI_Cancel

One thread must not cancel a request while that request is being serviced by another thread.

MPI_Probe, MPI_Iprobe

A call to MPI_Probe or MPI_Iprobe from one thread on a given communicator should not have a source rank and tags that match those of any other probes or receives on the same communicator. Otherwise, correct matching of message to probe call may not occur.

Collective Calls

Collective calls are matched on a communicator according to the order in which the calls are issued at each processor. All the processes on a given communicator must make the same collective call. You can avoid the effects of this restriction on the threads on a given processor by using a different communicator for each thread.

No process that belongs to the communicator may omit making a particular collective call; that is, none should be left "dangling."

Communicator Operations

Each of the communicator functions operates simultaneously with each of the noncommunicator functions, regardless of what the parameters are and of whether the functions are on the same or different communicators. However, if you are using multiple instances of the same communicator function on the same communicator, where all parameters are the same, it cannot be determined which threads belong to which resultant communicator. Therefore, when concurrent threads issue such calls, you must assure that the calls are synchronized in such a way that threads in different processes participating in the same communicator operation are grouped together. Do this either by using a different base communicator for each call or by making the calls in single-thread mode before actually using them within the separate threads.

Please note also these special situations:

• If you are using multiple instances of the same function with differing parameters and multiple threads, you must use different communicators. You must not use multiple instances of the same function on the same communicator with other differing parameters.

• When using splits with multiple instances of the same function with the same parameters, but with different threads at the split, you must use different communicators.

For example, suppose you wish to produce several communicators in different sets of threads by performing MPI_Comm_split on some base communicator. To ensure proper, thread-safe operation, you should replicate the base communicator via MPI_Comm_dup (in the root thread or in one thread) and then perform MPI_Comm_split on the resulting duplicate communicators.

• Do not free a communicator in one thread if it is still being used by another thread.

Error Handlers

When an error occurs as a result of an MPI call, the handler may not run on the same thread as the thread that made the error-raising call. In other words, you cannot assume that the error handler will execute in the local context of the thread that made the error-raising call. The error handler may be executed by another thread on the same process, distinct from the one that returns the error code. Therefore, you cannot rely on local variables for error handling in threads; instead, use global variables from the process.