Distributing S3L Arrays

Sun S3L distributes arrays in a block cyclic fashion. This means each array axis is partitioned into blocks of a certain block size and the blocks are distributed onto the processes in a cyclic fashion.

Block cyclic distribution is a superset of simple block distribution, a more commonly used array distribution scheme. Figure 2-4 through Figure 2-6 illustrate block and block cyclic distributions with a sample 8x8 array distributed onto a 2x2 process grid.

In Figure 2-4 and Figure 2-5, block size is set to 4 along both axes and the resulting blocks are distributed in pure block fashion. As a result, all the subgrid indices on any given process are contiguous along both axes.

The only difference between these two examples is that process grid ordering is column-major in Figure 2-4 and row-major in Figure 2-5.

Figure 2-4 An 8x8 S3L Array Distributed on a 2x2 Process Grid Using Pure Block Distribution: Column-Major Order

Figure 2-5 An 8x8 S3L Array Distribution on a 2x2 Process Grid Using Pure Block Distribution: Row-Major Ordering of Processes

Figure 2-6shows block cyclic distribution of the same array. In this example, the block size for the first axis is set to 4 and the block size for the second axis is set to 2.

Figure 2-6 An 8x8 S3L Array Distributed on a 2x2 Process Grid Using Block-Cyclic Distribution: Column-Major Order

When no part of an S3L array is distributed--that is, when all axes are local--all elements of the array are on a single process. By default, this is the process with MPI rank 0. The programmer can request that an undistributed array be allocated to a particular process via the S3L_declare_detailed routine.

Although the elements of an undistributed array are defined only on a single process, the S3L array handle enables all other processes to access the undistributed array.