Sun HPC ClusterTools 6 Software User's Guide
|
   
|
| Sun HPC ClusterTools 6 Software User's Guide
|
|
| C H A P T E R 5 |
|
Running Programs With mprun in Distributed Resource Management Systems |
This chapter describes the options to the mprun command that are used for distributed resource management, and provides instructions for each resource manager. It has four sections:
Call mprun from within the resource manager, as explained in Integration With Distributed Resource Management Systems. Use the -x flag to specify the resource manager, and the -np and -nr flags to specify the resources you need. In addition, the -Is flag selects the default CRE I/O environment, the -v flag produces verbose output, and the -J flag displays a fuller identification of each process.
Some mprun flags do not make sense for a batch job, and will cause the mprun request to be rejected if used with the -x flag. See Improper Flag Combinations for Batch Jobs.
These are the DRM integration options for mprun:
When launching mprun from a resource manager, the -Is option selects CRE's default I/O behavior, overriding the I/O behavior of the resource manager. You do not need this option when using any of these mprun flags: -I, -D, -N, -B, -n, -i, or -o.
Each of those flags already invokes CRE's default I/O behavior. You also do not need this option if you prefer to keep the resource manager's default I/O behavior.
Request the specified number of processes in a job. The default is 1 if you are using CRE. For other resource managers, the default is 0. . Use the argument 0 to specify that you want to start one process for each available CPU, based on your resource requirements.
When launching a multithreaded program, use the x threads syntax to specify the number of threads per process. Although the job requires a number of resources equal to numprocs multiplied by threads, only numprocs processes are started. The ranks are numbered from 0 to numprocs minus 1. The processes are allocated across nodes so that each node provides a number of CPUs equal to or greater than threads. If threading requirements cannot be met, the job fails and provides diagnostic messages.
A threads setting of 0 allocates the processes among all available resources. It is equivalent to the -Ns option.
The syntax -np numprocs is equivalent to the syntax -np numprocsx1. The default is -np 1x1.
If your batch job calls MPI_Comm_spawn(3SunMPI) or MPI_Comm_spawn_multiple(3SunMPI), be sure to use the -nr option to reserve the additional resources.
The -x option specifies the resource_manager. Supported resource_ managers are:
If you set a default resource manager in the hpc.conf(4) file, mprun is automatically launched with that resource manager and you don't have to use the -x option. To override the default, use the -x resource-manager flag.
Verbose output. Each interaction of the CRE environment with the resource manager is displayed in the output.
Do not use the following flags with the -x resource_manager flag; if you do, the mprun request will be rejected:
First reserve the number of resources by invoking the qsub command with the
-l option. The -l option specifies the number of nodes and the number of processes per node. For example, this command sequence reserves four nodes with four processes per node for the job myjob.sh:
Once you enter the PBS environment, you can launch an individual job or a series of jobs with mprun. Use the -x pbs option to the mprun command. The mprun command launches the job using the rankmap file produced by PBS and stored in the environment variable PBS_NODEFILE. The job ranks are children of PBS, not CRE.
You can run a CRE job within the PBS environment in two different ways:
|
1. Enter the PBS environment interactively with the -I option to qsub, and use the
-l option to reserve resources for the job.
The command sequence shown above enters the PBS environment and reserves one node with two processes for the job. Here is the output:
2. Launch the mprun command with the -x pbs option.
Here is an example that launches the hostname command with a verbose output:
The hostname program uses the rankmap specified by the PBS environment variable PBS_NODEFILE. The output shows the hostname program being run on ranks r0 and r1:
|
1. Write a script that calls mprun with the -x pbs option.
As described on page -x resource_manager, the -x flag identifies the resource manager that will be used for the job launched by mprun. In the following examples, the script is called myjob.csh. Here is an example of the script.
The line above launches the hostname program in verbose mode, using PBS as the resource manager.
2. Enter the PBS environment and use the -l option to qsub to reserve resources for the job.
Here is an example of how to use the -l option with the qsub command.
The command sequence shown above enters the PBS environment and reserves one node with two processes for the job that will be launched by the script named myjob.csh.
Here is the output to the script myjob.csh.
As you can see, because the mprun command was invoked with the -x pbs option, it calls the pbsrun command, which calls mpexec, which forks into two calls of the hostname program, one for each node.
|
1. Enter the LSF environment with the bsub command.
a. Use the -Is option to select interactive mode.
b. Use the -n option to reserve resources for the job.
c. Use the -q option to select the queue.
Here is an example that shows how to use the -q option.
The command sequence shown above enters the LSF environment in interactive mode, reserves 4 nodes, and selects the short queue. Here is the output:
Job <24559> is submitted to queue <short> <<Waiting for dispatch...>> <<Starting on burl-ct-v4>> burl-ct-v4 |
2. Use pam and mprun as shown below to start the parallel job.
pam requires the -g switch, which specifies the generic job launcher framework. mprun requires the -x lsf switch in order to specify that it is running under the LSF resource manager.
|
1. Write a script that calls mprun with the -x lsf option.
As described on page -x resource_manager, the -x flag identifies the resource manager that will be used for the job launched by mprun. Here is an example of the script.
The line above launches the hostname program in verbose mode, using LSF as the resource manager.
2. Enter the LSF environment with the bsub command.
a. Use the -n option to reserve resources for the job.
b. Use the -q option to select the queue.
c. Invoke the script. Make sure you precede it with pam -g.
burl-ct-v4-4 139 =>bsub -n 4 -q short pam -g myjob.csh Job <1152> is submitted to queue <short>. burl-ct-v4-4 140 => |
The command sequence shown above enters the LSF environment, reserves 4 nodes, selects the short queue, and invokes the script myjob.csh, which calls mprun.
You can launch MPI programs from within SGE in two different ways:
|
Note - Sun N1 Grid Engine 6 (N1GE6) is the supported version of Sun Grid Engine in Sun HPC ClusterTools 6. For the purposes of this manual, N1GE6 is referred to as SGE. |
Before you can use SGE with HPC ClusterTools, you need to set up the queue and parallel environment (PE) in SGE/N1GE. For information about how to set up the queue and the PE to work with HPC ClusterTools, refer to the Sun HPC ClusterTools 6 Software Administrator's Guide.
|
1. Enter the SGE environment with the qsh command.
a. Use the -pe option to reserve resources for the job.
b. Use the cre option to specify CRE as the parallel processing environment.
The command sequence shown above enters the SGE environment in interactive mode, reserves 2 nodes, and specifies CRE as the parallel processing environment. Here is the output from the command sequence:
waiting for interactive job to be scheduled ... Your interactive job 24 has been successfully scheduled. |
2. Enter the mprun command with the -x sge option.
The output shows the hostname program being run on ranks r0 and r1:
[r0: aout: qrsh, args: qrsh -inherit -V hpc-u2-7 /opt/SUNWhpc/lib/mpexec -x sge -- hostname] [r1: aout: qrsh, args: qrsh -inherit -V hpc-u2-6 /opt/SUNWhpc/lib/mpexec -x sge -- hostname] |
|
1. Write a script that calls mprun with the -x sge option.
As described on page -x resource_manager, the -x flag identifies the resource manager that will be used for the job launched by mprun. Here is an example of a script.
The line above launches the hostname program in verbose mode, using SGE as the resource manager.
2. Enter the SGE environment with the qsub command.
a. Use the -pe option to reserve resources for the job.
b. Use the -cre option to specify CRE as the parallel processing environment.
c. In the qsub syntax, use the script name instead of the program name.
Here is an example of how to use qsub with the script name.
The command sequence shown above enters the CRE environment, reserves 2 nodes, and invokes the script myjob.csh, which calls mprun. Here is the output:
This is all you need to do to run the job.
3. To display the output from the job, find the output file and display its contents.
a. Use the ls command to list the files into which the script has loaded the output.
This example uses the job number to identify the output files:
The file that contains the job's errors is named myjob.csh.e33. The file that contains the job's output has the name myjob.csh.o33.
b. To view the job's output, display the contents of the job's output file.
Continuing with the example above:
| Sun HPC ClusterTools 6 Software User's Guide
|
819-4131-10 |
|
Copyright © 2006, Sun Microsystems, Inc. All Rights Reserved.
To Run an Interactive Job in PBS