test

Sun S3L 3.0 Programming and Reference Guide

Chapter 6 Using Sun S3L

This chapter explains how to implement calls to S3L routines into your F77, F90, C or C++ program. The following topics are included:

Sun S3L documentation includes sample online programs that demonstrate how to call each Sun S3L routine. You are encouraged to experiment with these sample programs. Online man pages are also included for all Sun S3L routines. " Online Sample Code and Man Pages "explains how to find the program examples.

Creating a Program that Calls Sun S3L Routines

To use Sun S3L routines in a program:

  1. Place calls to Sun S3L routines into your code.

  2. Include the appropriate header file in each program unit that calls Sun S3L routines.

    See " Include the Sun S3L Header File"for details.

  3. Use the appropriate compiler command to compile your code; include the Sun S3L link switch on the command line.

    See " Compiling and Linking"for details.

The remainder of this section describes the steps listed above more fully.

Sun S3L requires the presence of the Sun Performance Library routines and its associated license file. This library is not installed with Sun S3L and other Sun HPC ClusterTools components. Instead, it is included as part of the following compiler suites:

Include the Sun S3L Header File

Place the appropriate include line at the top of any program unit that makes an S3L call. The correct include files are shown below for both C and Fortran language interfaces:

The first line allows the program to access the header file containing prototypes of the routines and defines the symbols and data types required by the interface. The second line includes the header file containing error codes the routines might return.

If the compiler cannot find the Sun S3L include file, verify that a path to the directory does exist. The standard path is 

/opt/SUNWhpc/includ/

If the file appears to be missing, consult your system administrator.

Compiling and Linking

Compile your program and link in Sun S3L (along with any other libraries it needs).

The link-line switch -ls3l does more than just link in Sun S3L subroutines. Depending on which compiler has been invoked, it also automatically links any other libraries needed to augment Sun S3L, greatly simplifying the link line.

Note -

The -dalign option is needed because libs3l and libsunperf libraries are compiled with it.

Executing Sun S3L Programs

Execute a program that has been linked with Sun S3L just as you would any other program compiled for running on a Sun HPC System.

To submit such an application to the LSF Batch system, use the LSF bsub command. For example,

% bsub -q hpc -n 4 hpc.job

submits the executable hpc.job to the batch queue hpc and requests that it run on four processors. The LSF Batch system will launch hpc.job as soon as it reaches the top of the queue and all required resources become available.

Refer to the Sun MPI 4.0 User's Guide: With LSF and the LSF Batch User's Guide for complete instructions on submitting Sun HPC jobs to the LSF Batch system.

To submit hpc.job to the Sun HPC Cluster Runtime Environment (CRE), use the mprun command. For example,

% mprun -np 4 hpc.job

submits hpc.job to the CRE and requests that it run on four processes.

Refer to the Sun MPI 4.0 User's Guide: With CRE for additional information.

The Sun S3L Safety Mechanism

The Sun S3L safety mechanism offers two basic features: It synchronizes the parallel processes so that you can pinpoint the area of code that generated an error. It also performs error checking and reports errors at a user-selectable level of detail.

Synchronization

When a Sun S3L application executes on multiple processes, the processes are generally running asynchronously with respect to one another. The Sun S3L safety mechanism provides an interface for explicitly synchronizing the processes to each Sun S3L call made by your code. It traps and reports errors, indicating when the errors occurred relative to the synchronization points.

Error Checking and Reporting

The safety mechanism can perform error checking and generate run-time error information at multiple levels of detail. You can turn safety checking on at any level during all or part of a program. One level checks for errors in the usage and arguments of the Sun S3L calls in your program; a more detailed level also checks for errors generated by internal Sun S3L routines. Examples of errors found and reported by the safety mechanism include the following:

Levels of Error Checking

The Sun S3L safety mechanism has four selectable levels: 0, 2, 5, and 9. These levels are defined in Table 6-1.

At levels 2, 5, and 9, some safety mechanism error messages are displayed at the terminal when you run the program; other information appears in the backtrace when you use a debugger such as Prism.

Table 6-1 S3L Safety Mechanism Levels

0 

Turns off the safety mechanism. Explicit synchronization and error checking are not performed. This level is appropriate for production runs of code that has already been thoroughly tested. 

2 

Detects potential race conditions in multithreaded S3L operations on parallel arrays. To avoid race conditions, an S3L function locks all parallel array handles in its argument list before proceeding. This safety level causes warning messages to be generated if more than one S3L function attempts to use the same parallel array at the same time. 

5 

Detects and reports all level-2 errors. In addition, level 5 performs explicit synchronization before and after each call and locates each error with respect to the synchronization points. This safety level is appropriate during program development or during runs for which a small performance penalty can be tolerated. 

9 

Checks for and reports all level 2 and level 5 errors, as well as errors generated by lower levels of code that were called from within S3L. Performs explicit synchronization in these lower levels of code and locates each error with respect to the synchronization points. This level performs all implemented error checking and exacts a very high performance price. It is appropriate for detailed debugging when a problem occurs. 

Selecting a Safety Mechanism Level

You can select the desired S3L safety mechanism level in either of two ways:

These methods are described in " Setting the Sun S3L Safety Environment Variable"and " Setting the Safety Level from Within a Program".

Setting the Sun S3L Safety Environment Variable

The S3L_SAFETY environment variable takes a single argument, which can be the integer 0, 2, 5, or 9. For example, to select the highest level, enter:

% setenv S3L_SAFETY 9

One advantage of using the S3L_SAFETY environment variable is that you can set or change the safety level without recompiling your code.

Setting the Safety Level from Within a Program

To set the Sun S3L safety level from within your program, include the following subroutine call. Specify the desired level in the integer argument n:

To see what Sun S3L safety level is currently in effect, include the following call. Again, specify the level of interest in the integer argument n:

The advantage of using these calls from within a program is that you can set or obtain the safety level at any point within your code. However, you must recompile the code each time you change these calls.

Online Sample Code and Man Pages

Sample Code Directories

The online sample programs are located in subdirectories of the S3L examples directory. Separate C and F77 versions are provided. The generic relative path for these examples is

examples/s3l/operation_class[-language_suffix]/example_name.language

where examples/s3l is installed in a site-specific location.

operation_class is the name of the general class of Sun S3L routines that are illustrated by the example.

The -language_suffix is used to denote F77 implementations. Examples implemented in C do not include the -language_suffix.

example_name.language is the name given to the example. The language extension is .c, or .f. For example,

examples/s3l/dense_matrix_ops-f/outer_prod.f

is the F77 version of a program example that illustrates use of s3l_outer_prod routines. The equivalent examples for C applications is

examples/s3l/dense_matrix_ops/outer_prod.c

Compiling and Running the Examples

Each example subdirectory has a makefile. Each makefile references the file ../Make.simple. If you are copying the example sources and makefiles to one of your own subdirectories, you should also copy Make.simple to your subdirectory's parent directory. Make.simple contains definitions of compilers, compiler flags and other variables that are needed to compile and run the examples. Note that the compiler flags in this file will not provide you with highly optimized executables. Information on optimization flags is best obtained from the documentation for the compiler of interest.

Each makefile has several targets that are meant to simplify the compilation and execution of examples. If you want to compile the source codes and create all executables in a particular example directory, use the command make.

If you wish to run the executables, enter make run. This command will also perform any necessary compilation and linking steps, so you need not issue make before entering make run.

By default, your executables will be run on two processes. You can change this by specifying the NPROCS variable on the command line. For example, 

% make run NPROCS=4

will start your runs on four processes.

Executables and object files can be deleted by make clean.

Man Pages

To read the online man page for a Sun S3L routine, enter

% man routine_name

Chapter 7, Sun S3L Toolkit Routines and Chapter 8, Sun S3L Core Library Functions also describe the Sun S3L routines. Chapter 7, Sun S3L Toolkit Routines covers the set of toolkit routines and Chapter 8, Sun S3L Core Library Functions describes the core (computational) routines.