Introduction to Sun HPC ClusterTools Software

C H A P T E R 1

Sun HPC ClusterTools 8.2.1c software is a set of parallel development tools that extend the Sun network computing solutions to high-end distributed-memory applications. This chapter summarizes its required configuration and principal components. It contains the following sections:

Supported Configurations

Open Runtime Environment (ORTE)

Integration With Distributed Resource Management Systems

Open MPI Features

Debugging With TotalView

Performance Analysis With Sun Studio Analyzer

Communications Failover Feature In Multi-Rail Infiniband Configurations

Supported Configurations

Sun HPC ClusterTools 8.2.1c software can be used in any of the following operating system environments:

Solaris 10 Operation System (Solaris 10 OS)

Red Hat Linux versions 5 (RHEL)

SUSE Linux versions 10 (SLES)

CentOS 5.3 Linux

OpenSolaris 2009.06

Sun HPC ClusterTools 8.2.1c software supports the following compilers in the Solaris OS and Linux environments:

Solaris OS

Sun Studio 10, 11, 12, and 12U1 C, C++, and Fortran

Linux

Sun Studio 12U1 C, C++, and Fortran

gcc 3.3.3, 3.4.6, 4.1.2

Intel 11.0 20081105

PGI 7.1-4

Pathscale 3.2

Sun HPC ClusterTools 8.2.1c software can run MPI jobs of up to 4096 processes on as many as 1024 nodes. It also provides support for spawning MPI processes.

The Sun HPC ClusterTools 8.2.1c software runs on clusters connected by any TCP/IP-capable interconnect, such as high-speed Ethernet, Gigabit Ethernet, Infiniband, and Myrinet MX.

Open Runtime Environment (ORTE)

Sun HPC ClusterTools 8.2.1c is based on the Open MPI message-passing interface. Open MPI operates using the Open Runtime Environment (ORTE). ORTE starts jobs and provides some status information.

The Open MPI mpirun and mpiexec commands are actually symbolic links to the orterun command. All three commands perform the same function, which is to launch MPI jobs.

ORTE is compatible with a number of other launchers, including rsh/ssh, Sun Grid Engine, PBS Professional, and open PBS (Torque).

Each of ORTE’s primary operations is summarized in the sections that follow. Subsequent chapters contain the procedures.

Executing Programs With `mpirun`

Sun HPC ClusterTools 8.2.1c software can start both serial and parallel jobs. The syntax and use of mpirun are described in Chapter 5.

Integration With Distributed Resource Management Systems

Sun HPC ClusterTools 8.2.1c software provides integration facilities with three select distributed resource management (DRM) systems. These systems provide proper resource allocation, parallel job control and monitoring, and proper job accounting. They are:

Sun Grid Engine Version 6.1 and 6.2 software

Altair PBS Professional 9.2

Cluster Resources Torque 2.3

Note - Open MPI itself supports other third-party launchers supported by Open MPI, such as SLURM. However, these launchers are currently not supported in Sun HPC ClusterTools software. To use these other third-party launchers, you must download the Open MPI source, compile, and link with the libraries for the launchers.

You can launch parallel jobs directly from these distributed resource management systems. The DRM interacts closely with ORTE for proper resource description and with the multiple processes comprising the requested parallel job.

For a description of the scalable and open architecture of the DRM integration facilities, see How the Open MPI Environment Is Integrated With Distributed Resource Management Systems. For instructions, see Chapter 6.

Open MPI Features

Open MPI is a highly optimized version of the Message Passing Interface (MPI) communications library. It implements all of the MPI 1.2 Standard and the MPI 2.0 Standard. Its highlights are:

Integration with the Open Runtime Environment (ORTE)

Support for MPI I/O

Seamless use of different network protocols; for example, code compiled on a Sun HPC cluster that has a fast Ethernet network can be run without change on a cluster that has an Infiniband network

Multiprotocol support so that MPI picks the fastest available medium for each type of connection (such as shared memory, fast Ethernet, Infiniband)

Communication via shared memory for fast performance

Optimized collectives for multiprocessors and clusters of multiprocessors

Full F77, C, and C⁺⁺ support, and basic F90 support

Debugging With TotalView

TotalView is a third-party multiprocess debugger from TotalView Technologies (formerly Etnus) that runs on many platforms. Support for using the TotalView debugger on Open MPI applications includes:

Making Sun HPC ClusterTools software compatible with the TotalView debugger

Allowing Open MPI jobs to be debugged by TotalView using the Sun Grid Engine or the Portable Batch System (PBS)

Allowing multiple instantiations of TotalView on a single cluster

Supporting TotalView in Sun HPC ClusterTools software

Refer to the TotalView documentation at http://www.totalviewtech.com for more information about using TotalView.

In addition, the Open MPI Frequently Asked Questions (FAQ) contains information about how to use the TotalView debugger with Open MPI. This information is available at:

http://www.open-mpi.org/faq/?category=running#run-with-tv

Performance Analysis With Sun Studio Analyzer

Analyzer is a performance analysis tool that is bundled with the Sun Studio compilers. It is available for free download at:

http://developers.sun.com/sunstudio/downloads

Analyzer offers the following capabilities:

Performance profiling of both MPI calls and user application code.

Tracing of MPI function calls and messages.

Metrics based on a variety of conditions, such as: run time, hardware counters, memory allocations and leaks, MPI messages and bytes, MPI "wait" and "work" time.

Attribution of metrics to user functions, callstacks, and source lines.

Timeline displays of program execution, including message lines.

Easy-to-use interface to construct general displays of statistical aggregation of performance data.

Filtering to examine subsets of MPI trace data.

For example, with Analyzer, you could replace the following command line:

% mpirun -np 4 ./a.out

with the following command lines:

% collect -M CT mpirun -np 4 -- ./a.out% analyzer test.1.er

The -M CT command turns on MPI tracing for ClusterTools.

Note - For older versions of collect, you may specify -M CT8.2, even to trace CT8.2.1c runs. Use collect without any arguments to see what values of -M your version of collect recognizes.

To use the collect command, you must have the Sun Studio tools in your execution path.

The -- symbol marks the executable that will be launched by mpirun.

The analyzer command starts up the Analyzer program, including a graphical user interface.

The test.1.er argument is the default experiment name generated by the collect command.

For more information on using Analyzer with MPI codes, see the web page at:

http://wikis.sun.com/display/MPIAnalyzer/Home

Analyzer documentation is available in Sun Studio manuals and at:

http://developers.sun.com/sunstudio/overview/topics/analyzing.jsp

Communications Failover Feature In Multi-Rail Infiniband Configurations

Sun HPC ClusterTools 8.2.1c software includes a failover feature for handling network failures in multi-rail Infiniband configurations. When a completion error is detected on a given rail, the failover software maps out that rail and routes future traffic through other rails available to the process.

The failover feature supports Open MPI openib BTL (Open Fabrics User Verbs) for the communications layer.

Note - The failover feature does not function with uDAPL or IPoIB.