Allocating Arrays in Shared Memory

Sun S3L supports the allocation of S3L arrays in shared memory. When an MPI program runs on a cluster of nodes, processes collocated on the same node can allocate their local array parts in that node's shared memory. Storing array sections in shared memory allows collocated processes to access each others array elements without going through MPI. This can yield significant performance improvements.

A special case of this would be an MPI application running on a single node. In this case, the entire S3L array could be allocated in shared memory.

Several Sun S3L functions are optimized for shared-memory accesses. That is, they employ different, more efficient algorithms when S3L arrays have been allocated in shared memory. These functions include the single and multidimensional parallel FFTs, as well as array transpose and sparse solver routines.

Use the S3L_declare or S3L_declare_detailed functions to allocate a parallel array in shared memory. For the type of allocation, specify either:

• S3L_USE_MMAP, which uses mmap() to allocate the S3L array

• S3L_USE_SHMGET, which uses shmget() (System V shared memory) to obtain Intimate Shared Memory. This is a form of shared memory in which data is always kept resident in physical memory and is never paged out.

  ---

  Note -

  Since physical memory is locked up by an application using Intimate Shared Memory, S3L_USE_SHMGET should only be used when the S3L array is small and the system is not being shared with other users.

  ---