Sun S3L 3.0 Programming and Reference Guide

Declaring Parallel Arrays

The Sun S3L toolkit functions described in this section share a common purpose--they all enable you to define parallel arrays that can then be operated on by other Sun S3L routines. Each of these routines returns an S3L array handle, which the application uses to reference the parallel array in subsequent S3L calls.

Each array declaring routine is described separately below.

Declare a parallel array - See "S3L_declare ".

Declare a parallel array with detailed control over attributes - See "S3L_declare_detailed "

Define a parallel array - "S3L_DefineArray ".

Sun S3L also provides a routine for declaring a sparse matrix. This routine, called, S3L_declare_sparse is described in "S3L_declare_sparse ".

`S3L_declare`

Description

S3L_declare creates an S3L array handle that describes an S3L parallel array . It supports calling arguments that enable the user to specify

the array's rank (number of dimensions)

the extent of each axis

the array's data type

which axes, if any, will be distributed locally

how memory will be allocated for the array

Based on the argument-supplied specifications, a process grid size is internaly determined to distribute the array as evenly as possible.

Note -

An array subgrid is the set of array elements that is allocated to a particular process.

The axis_is_local argument specifies which array axes (if any) will be local to the process. It consists of an integer vector whose length is at least equal to the rank (number of dimensions) of the array. Each element of the vector indicates whether the corresponding axis is local or not: 1 = local, 0 = not local.

When axis_is_local is ignored, all array axes except the last will be local. The last axis will be block distributed.

For greater control over array distribution, use S3L_declare_detailed().

Upon successful completion, S3L_declare returns an S3L array handle, which subsequent S3L calls can use as an argument to gain access to that array.

Syntax

The C and Fortran syntax for S3L_declare are shown below.

C/C++ Syntax

Example 7-5

#include <s3l/s3l-c.h>
#include <s3l/s3l_errno-c.h>
int
S3L_declare(A, rank, extents, type, axis_is_local, atype)
    S3L_array_t        *A
    int                rank
    int                *extents
    S3L_data_type      type
    S3L_boolean_t      *axis_is_local
    S3L_alloc_type     atype

F77/F90 Syntax

Example 7-6

include `s3l/s3l-f.h'
include `s3l/s3l_errno-f.h'
subroutine
S3L_declare(A, rank, extents, type, axis_is_local, atype, ier)
    integer*8           A
    integer*4          rank
    integer*4          extents(*)
    integer*4          type
    integer*4          axis_is_local(*)
    integer*4          atype
    integer*4          ier

Input

extents - An integer vector whose length is equal to the number of dimensions in the array. Each element in extents specifies the extent of the corresponding array axis. Note that axis indexing is zero-based for the C interface and one-based for the Fortran interface, as follows:
- When called from a C or C++ application, the first element of extents corresponds to axis 0, the second element to axis 1, and so forth.
- When called from an F77 or F90 application, the first element corresponds to axis 1, the second to axis 2, and so forth.

type - Specifies the array's data type; this must be a type supported by Sun S3L. See Chapter 4, Sun S3L Data Types for a complete list of supported data types.

axis_is_local - An integer vector whose length equals the array's rank. Each element of axis_is_local controls the distribution of the corresponding array axis as follows:
- If axis_is_local[i]= 0, axis[i] of the array will be block-distributed along axis [i] of the process grid.
- If axis_is_local[i]= 1, axis[i] will not be distributed.
  
  If axis_is_local is NULL (C/C++) or if its first integer value is negative (F77/F90), this argument will be ignored.

atype - Use one of the following predefined values to specify how the array will be allocated:
- S3L_USE_MALLOC - Uses malloc() to allocate the array subgrids.
- S3L_USE_MEMALIGN64 - Uses memalign() to allocate the array subgrids and to align them on 64-bit boundaries.
- S3L_USE_MMAP- Uses mmap() to allocate the array subgrids. Array subgrids on the same node will be in shared memory.
- S3L_USE_SHMGET - Uses shmget() to allocate the array subgrids. Array subgrids on the same node will be in intimate shared memory.

Output

S3L_declare uses the following arguments for output:

A - S3L_declare returns the array handle in A.

ier (Fortran only) - When called from a Fortran program, S3L_declare_detailed returns error status in ier.

Error Handling

On successful completion, S3L_declare returns S3L_SUCCESS.

S3L_declare applies verious checks to the arrays it accepts as arguments. If an array argument fails any of these checks, the function returns an error code indicating the kind of error that was detected and terminates. See Appendix A, S3L Array Checking Errors of this manual for a list of these error codes.

In addition, the following conditions will cause S3L_declare to terminate and return the associated error value:

S3L_ERR_ARG_RANK - The rank specified is invalid.

S3L_ERR_ARG_EXTENTS - One or more of the array extents is less than 1.

S3L_ERR_ARG_BLKSIZE - One or more blocksizes is less than 1.

S3L_ERR_ARG_DISTTYPE - axis_is_local has one or more invalid values. See the description of axis_is_local in the Input section for details.

Notes

When S3L_USE_MMAP or S3L_USE_SHMGET is used on a 32-bit platform, the part of an S3L array owned by a single SMP cannot exceed 2 gigabytes.

When S3L_USE_MALLOC or S3L_USE_MEMALIGN64 is used, the part of the array owned by any single process can not exceed 2 gigabytes.

If these size restrictions are violated, an S3L_ERR_MEMALLOC will be returned. On 64-bit platforms, the upper bound is equal to the system's maximum available memory.

Examples

../examples/s3l/transpose/ex_trans1.c
../examples/s3l/grade-f/ex_grade.f

Related Functions

S3L_declare_detailed(3)
S3L_free(3)

`S3L_declare_detailed`

Description

This subroutine offers the same functionality as S3L_declare, but supports the additional input argument, addr_a, which gives the user additional control over array distribution.

Syntax

The C and Fortran syntax for S3L_declare_detailed are shown below.

C/C++ Syntax

Example 7-7

#include <s3l/s3l-c.h>
#include <s3l/s3l_errno-c.h>
int
S3L_declare_detailed(A, addr_a, rank, extents, type, blocksizes, proc_src,
axis_is_local, pgrid, atype)
    S3L_array_t        *A
    void               *addr_a
    int                rank
    int                *extents
    S3L_data_type      type
    int                *blocksizes
    int                *proc_src
    S3L_boolean_t      *axis_is_local
    S3L_pgrid_t        pgrid
    S3L_alloc_type     atype

F77/F90 Syntax

Example 7-8

include `s3l/s3l-f.h'
include `s3l/s3l_errno-f.h'
subroutine
S3L_declare_detailed(A, addr_a, rank, extents, type, blocksizes, proc_src,
axis_is_local, pgrid, atype, ier)
    integer*8          A
    <type>             array(1)
    pointer            (addrr_a,array)
    integer*4          rank
    integer*4          extents(*)
    integer*8          type
    integer*4          *blocksizes
    integer*4          *proc_src
    integer*4          axis_is_local(*)
    integer*8          pgrid
    integer*4          atype
    integer*4          ier

where <type> is one of: integer*4, integer*8, real*4, real*8, complex*8, or complex*16.

Input

S3L_declare_detailed accepts the following arguments as input:

addr_a - If the atype argument is set to S3L_DONOT_ALLOCATE, addr_a is taken as the starting address of the local (per process) portion of the parallel array A. If atype is not equal to S3L_DONOT_ALLOCATE, addr_a will be ignored.

rank - Specifies the number of dimensions the array will have. The range of legal values for rank is 1 <= rank <= 31.

extents - An integer vector whose length is equal to the number of dimensions in the array. Each element in extents specifies the extent of the corresponding array axis. Note that axis indexing is zero-based for the C interface and one-based for the Fortran interface, as follows:
- When called from a C or C++ application, the first element of extents corresponds to axis 0, the second element to axis 1, and so forth.
- When called from an F77 or F90 application, the first vector element corresponds to axis 1, the second to axis 2, and so forth.

type - Specifies the array's data type; this must be a type supported by Sun S3L. See Chapter 4, Sun S3L Data Types for a complete list of supported data types.

blocksizes - Specifies the blocksize to be used in a block cyclic distribution along each axis. If blocksizes is NULL (C/C++) or if its first element is equal to -1 (F77/F90), default blocksizes will be chosen by the system.

proc_src - Vector of length at least equal to the rank. The indices of the processes contain the start of the array--that is, the first element along the particular axis. If this argument is a NULL pointer (C/C++) or if its first element is less than zero (F77/F90), default values will be used. Along each axis, the process whose process grid coordinate along that axis is equal to 0 contains the first array element. If present, the pgrid argument (process grid) should also be present. Otherwise an error code will be returned.

axis_is_local - An integer vector whose length equals the number of dimensions in the array. Each element of axis_is_local controls the distribution of the corresponding array axis as follows:
- If axis_is_local[i]= 0, axis[i] of the array will be block-distributed along axis [i] of the process grid.
- If axis_is_local[i]= 1, axis[i] will not be distributed.
  
  The axis_is_local argument is used only if a pgrid is not specified. If it is NULL (C/C++) or if its first integer value is negative (F77/F90), axis_is_local will be ignored.
  
  Note -
  A process grid is the array of processes onto which the data is distribued.

pgrid- An S3L process grid handle that was obtained by calling either S3L_set_process_grid or S3L_get_attribute. If this argument is NULL (C/C++) or is equal to zero (F77/F90), S3L will choose an appropriate pgrid for the array.

atype - Use one of the following predefined values to specify how the array will be allocated:
- S3L_USE_MALLOC - Uses malloc() to allocate the array subgrids.
- S3L_USE_MEMALIGN64 - Uses memalign() to allocate the array subgrids and to align them on 64-bit boundaries.
- S3L_USE_MMAP- Uses mmap() to allocate the array subgrids. Array subgrids on the same SMP will be in shared memory.
- S3L_USE_SHMGET - Uses shmget() to allocate the array subgrids. Array subgrids on the same SMP will be in shared memory.

Output

A - S3L_declare_detailed returns the array handle in A.

ier (Fortran only) - When called from a Fortran program, S3L_declare_detailed returns error status in ier.

Error Handling

On successful completion, S3L_declare_detailed returns S3L_SUCCESS.

S3L_declare_detailed applies various checks to the arrays it accepts as arguments. If an array argument fails any of these checks, the function returns an error code indicating the kind of error that was detected and terminates. See Appendix A, S3L Array Checking Errors of this manual for a list of these error codes.

In addition, the following conditions will cause S3L_declare_detailed to terminate and return the associated error value:

S3L_ERR_ARG_RANK - The rank specified is invalid in one of the following ways:
- If called from a C/C++ program, it is negative or greater than 31
- If called from an F77/F90 program, it is zero or greater than 32.

S3L_ERR_ARG_EXTENTS - One or more of the array extents is less than 1.

S3L_ERR_ARG_BLKSIZE - One or more blocksizes is less than 1.

S3L_ERR_ARG_DISTTYPE - axis_is_local has one or more invalid values. See the description of axis_is_local in the Input section for details.

Notes

When S3L_USE_MMAP or S3L_USE_SHMGET is used on a 32-bit platform, the part of an S3L array owned by a single SMP cannot exceed 2 gigabytes.

When S3L_USE_MALLOC or S3L_USE_MEMALIGN64 is used, the part of the array owned by any single process cannot exceed 2 gigabytes.

If these size restrictions are violated, an S3L_ERR_MEMALLOC will be returned. On 64-bit platforms, the upper bound is equal to the system's maximum available memory.

Examples

../examples/s3l/utils/copy_array.c
../examples/s3l/utils-f/copy_array.f
../examples/s3l/utils/get_attribute.c
../examples/s3l/utils-f/get_attribute.f
../examples/s3l/utils/scalapack_conv.c
../examples/s3l/utils-f/scalapack_conv.f

Related Functions

S3L_declare(3)
S3L_free(3)
S3L_get_process_grid(3)
S3L_set_process_grid(3)

`S3L_DefineArray`

Description

S3L_DefineArray associates an internal S3L array handle to a user-distributed parallel array. The array must be distributed in such a manner that it can be expressed as a block cyclic distribution. The array handle returned by S3L_DefineArray can then be used in subsequent calls by Sun MPI programs to S3L functions.

S3L_DefineArray does not allocate the memory required to store the local (process specific) part on the array. The user must allocate sufficient memory on each process to hold the local part of the parallel array before calling this function.

Syntax

The C and Fortran syntax for S3L_DefineArray are shown below.

C/C++ Syntax

Example 7-9

#include <s3l/s3l-c.h>
#include <s3l/s3l_errno-c.h>
S3L_array_t
S3L_DefineArray(address, rank, type, extents, blocks,
sprocs, p_ext)
    void               address
    int                rank
    int                type
    int                *extents
    int                *blocks
    int                *sprocs
    int                *p_ext

F77/F90 Syntax

Example 7-10

include `s3l/s3l-f.h'
include `s3l/s3l_errno-f.h'
integer*8 function
S3L_DefineArray(address,
rank, type, extents, blocks, sprocs, p_ext)
    <type>             array(1)
    integer*4          rank
    pointer            (addrr_a,array)
    integer*4          rank
    integer*4          type
    integer*4          extents(*)
    integer*4          blocks(*)
    integer*4          sprocs(*)
    integer*4          p_ext(*)

Input

S3L_DefineArray accepts the following arguments as input:

address - The starting address of the local (within the process) portion of a parallel array. In C, the user must allocate this local memory (for example, via the malloc function). In F77, the address is defined as a pointer to a local storage area.
rank - Specifies the number of dimensions the array will have. The range of legal values for rank is 1 <= rank <= 31.
type - Denotes the parallel array's data type. In C, it is a variable of type S3L_data_type. In F77, it is a variable of type integer*4.
extents - Specifies the extents of the parallel array.
blocks - Specifies the blocksizes of the block cyclic distribution of the parallel array along each axis. Note that, block cyclic distribution is a general classification that includes other distribution types, such as CYCLIC(1) and BLOCK(n) distribution.
sprocs - The starting processes of the block cyclic array distributions. If sprocs[i] = j, block 0 of the block cyclic distribution of the array along axis [i] is located in process j.
p_ext - The extents of the process grid upon which the array is distributed. If, for example, p_ext[0] = 2, p_ext[1] = 3, p_ext[2] = 4, the three-dimensional parallel array will be distributed on a 2 x 3 x 4 process grid.

Note that process ordering within the process grid is always column major (F77 major) and that the product of the extents of the process grid must equal the total number of processes participating in the computation (that is, must equal the size of MPI_COMM_WORLD).

Error Handling

On success, S3L_DefineArray returns an S3L array handle that can be used for subsequent calls to other Sun S3L functions.

On error, it returns 0.

Examples

../examples/s3l/api
../examples/s3l/api-f

Related Functions

S3L_UnDefineArray(3)