| Sun HPC ClusterTools 7.1 Software User’s Guide |    
|
| Sun HPC ClusterTools 7.1 Software User’s Guide |
|
| C H A P T E R 5 |
|
Running Programs With mpirun in Distributed Resource Management Systems |
This chapter describes the options to the mpirun command that are used for distributed resource management, and provides instructions for each resource manager. It contains the following sections:
ORTE is compatible with a number of other launchers, including rsh/ssh, Sun Grid Engine, and PBS.
To see whether your Open MPI installation has been configured for use with the third-party resource manager you want to use, issue the ompi_info command and pipe the output to grep. The following examples show how to use ompi_info to check for the desired third-party resource manager.
To see whether your Open MPI installation has been configured to use the rsh/ssh launcher:
To see whether your Open MPI installation has been configured to use the PBS/Torque launcher:
% ompi_info | grep tm MCA ras: tm (MCA v1.0, API v1.3, Component v1.2) MCA pls: tm (MCA v1.0, API v1.3, Component v1.2) |
To see whether your Open MPI installation has been configured to use Sun Grid Engine:
% ompi_info | grep gridengine MCA ras: gridengine (MCA v1.0, API v1.3, Component v1.2) MCA pls: gridengine (MCA v1.0, API v1.3, Component v1.2) |
If your Open MPI environment is set up to include PBS, Open MPI automatically detects when mpirun is running within PBS, and will execute properly.
First reserve the number of resources by invoking the qsub command with the
-l option. The -l option specifies the number of nodes and the number of processes per node. For example, this command sequence reserves four nodes with four processes per node for the job myjob.sh:
When you enter the PBS environment, you can launch an individual job or a series of jobs with mpirun. The mpirun command launches the job using the nodes and processes information from PBS.The resource information is accessed using the tm calls provided by PBS; hence, tm is the name used to identify the module in ORTE. The job ranks are children of PBS, not ORTE.
You can run an ORTE job within the PBS environment in two different ways: interactive and scripted.
|
1. Enter the PBS environment interactively with the -I option to qsub, and use the
-l option to reserve resources for the job.
The command sequence shown above enters the PBS environment and reserves one node called mynode with two processes for the job. Here is the output:
qsub: waiting for job 20.mynode to start qsub: job 20.mynode ready Sun Microsystems Inc. SunOS 5.10 Generic January 2005 pbs% |
Here is an example that launches the hostname command with a verbose output:
The output shows the hostname program being run on ranks r0 and r1:
The following example shows the debugging output specific to the tm module using the MCA parameter pls_tm_debug.
|
1. Write a script that calls mpirun.
In the following examples, the script is called myjob.csh. The system is called mynode. Here is an example of the script.
2. Enter the PBS environment and use the -l option to qsub to reserve resources for the job.
Here is an example of how to use the -l option with the qsub command.
This command enters the PBS environment and reserves one node with two processes for the job that will be launched by the script named myjob.csh.
Here is the output to the script myjob.csh.
After the job finishes, it generates two output files:
As you can see, the pbsrun command calls mpirun, which forks into two calls of the hostname program, one for each node.
Sun Grid Engine 6 is the supported version of Sun Grid Engine in Sun HPC ClusterTools 7.1.
Before you can run parallel jobs, make sure that you have defined the parallel environment and queue before running the job.
A PE needs to be defined for all the queues in the Sun Grid Engine cluster to be used as ORTE nodes. Each ORTE node should be installed as a Sun Grid Engine execution host. To allow the ORTE to submit a job from any ORTE node, configure each ORTE node as a submit host in Sun Grid Engine.
Each execution host must be configured with a default queue. In addition, the default queue set must have the same number of slots as the number of processors on the hosts.
To display a list of available PEs (parallel environments), type the following:
To define a new PE, you must have Sun Grid Engine manager or operator privileges. Use a text editor to modify a template for the PE. The following example creates a PE named orte.
To modify an existing PE, use this command to invoke the default editor:
To show a particular PE that has been defined, type this command:
The value NONE in user_lists and xuser_lists mean enable everybody and exclude nobody.
The value of control_slaves must be TRUE; otherwise, qrsh exits with an error message.
The value of job_is_first_task must be FALSE or the job launcher consumes a slot. In other words, mpirun itself will count as one of the slots and the job will fail, because only n-1 processes will start.
To show all the defined queues, type the following command:
The queue all.q is set up by default in Sun Grid Engine.
To configure the orte PE from the example in the previous section to the existing queue, type the following:
You must have Sun Grid Engine manager or operator privileges to use this command.
There are two ways to submit jobs under Sun Grid Engine integration: interactive mode and batch mode. The instructions in this section describe how to submit jobs in batch mode. For information about how to use interactive mode, see Chapter 4.
Before you submit a job, you must have your DISPLAY environment variable set so that the interactive window will appear on your desktop, if you have not already done so.
For example, if you are working in the C shell, type the following command:
1. Create the script. In this example, mpirun is embedded within a script to qsub.
| Note - The --mca pls_gridengine_debug 100 setting is used in this example only to show that Sun SGE is being used. This would not be needed for normal operation. |
2. Next, source the Sun Grid Engine environment variables from a settings.csh file where $SGE_ROOT is set to /opt/sge:.
3. To start the batch (or scripted) job, specify the parallel environment, slot number and the user executable.
Since this is submitted as a batch job, you would not expect to see output at the terminal. If no indication is given for where the output should go, Sun Grid Engine redirects to your home directory and creates <job_name>.o<job_number>.
| Note - Before you can use the parallel environment, make sure that you have set it up before running the job. See Defining Parallel Environment (PE) and Queue for more information. |
The job creates the output files. The file name with the format name_of_job.ojob_id contains the standard output. The file name with the format name_of_job.ejob_id contains the error output. If the job executes normally, the error output files will be empty.
The following example lists the files produced by a job called sge.csh with the job ID number 866:
By default, the output files are located in your home directory, but you can use Sun Grid Engine software to change the location of the files, if desired.
where job-number is the number of the job you want to delete.
For more information about Sun Grid Engine commands, refer to the Sun Grid Engine documentation.
For more information about using the mpirun command to perform batch processing, see the following:
| Sun HPC ClusterTools 7.1 Software User’s Guide | 819-7480-11 |
|
To Check for rsh/ssh