Shared-Memory Point-to-Point Message Passing

The size of each shared-memory buffer is fixed at 1 Kbyte. Most other quantities in shared-memory message passing are settable with MPI environment variables.

A short message, at most MPI_SHM_SHORTMSGSIZE bytes long, is fit into one postbox and no buffers are used. Above that size, message data is written into buffers and controlled by postboxes.

Only starting at MPI_SHM_PIPESTART bytes, however, are multiple postboxes used, which is known as pipelining. The amount of buffer data controlled by any one postbox is at most MPI_SHM_PIPESIZE bytes. By default, MPI_SHM_PIPESTART is well below MPI_SHM_PIPESIZE. For the smallest pipelined messages, then, a message is broken roughly into two, and each of two postboxes controls roughly half the message.

Above MPI_SHM_CYCLESTART bytes, messages are fed cyclically through two sets of buffers, each set of size MPI_SHM_CYCLESIZE bytes. During a cyclic transfer, the footprint of the message in shared memory buffers is 2*MPI_SHM_CYCLESIZE bytes.

The postbox area consists of MPI_SHM_NUMPOSTBOX postboxes per connection. By default, each connection has its own pool of buffers, each pool of size MPI_SHM_CPOOLSIZE bytes.

By setting MPI_SHM_SBPOOLSIZE, users may specify that each sender has a pool of buffers, of MPI_SHM_SBPOOLSIZE bytes each, to be shared among its various connections. If MPI_SHM_CPOOLSIZE is also set, then any one connection may consume only that many bytes from its send-buffer pool at any one time.

Memory Considerations

In all, the size of the shared-memory area devoted to point-to-point messages is

n * ( n - 1
)
  * (
      MPI_SHM_NUMPOSTBOX *
( 64 + MPI_SHM_SHORTMSGSIZE )
    + MPI_SHM_CPOOLSIZE
    )

bytes when per-connection pools are used (that is, when MPI_SHM_SBPOOLSIZE is not set) and

n * ( n - 1
) * MPI_SHM_NUMPOSTBOX *
( 64 + MPI_SHM_SHORTMSGSIZE )
+
n * MPI_SHM_SBPOOLSIZE

bytes when per-sender pools are used (that is, when MPI_SHM_SBPOOLSIZE is set).

Cyclic message passing limits the size of shared memory that is needed to transfer even arbitrarily large messages.