test

Sun HPC ClusterTools 3.0 Administrator's Guide: With CRE

mpadmin: Administration Interface

The CRE provides an interactive command interface, mpadmin, which you can use to administer your Sun HPC cluster. It must be invoked by root.

This section explains how to use mpadmin to perform the following administrative tasks:

These descriptions are preceded by an introduction of the mpadmin command interface, which is presented in "Introduction to mpadmin".

Note -

mpadmin offers many more capabilities than are described in this section. See Chapter 6 for a more comprehensive description of mpadmin.

Introduction to mpadmin

mpadmin Syntax

The mpadmin command has the following syntax.

# mpadmin [-c command] [-f filename] [-h] [-q] [-s cluster_name] [-V]

When you invoke mpadmin with the -c, -h, or -V options, it performs the requested operation and returns to the shell level.

When you invoke mpadmin with any of the other options (-f, -q, or -s), it performs the specified operation and then displays an mpadmin prompt, indicating that it is in the interactive mode. In this mode, you can execute any number of mpadmin commands until you quit the interactive session.

When you invoke mpadmin without any options, it goes immediately into the interactive mode, displaying an mpadmin prompt.

The mpadmin command-line options are summarized in Table 3-1 and more fully following the table.

Table 3-1 mpadmin Options

Option 

Description 

-c command

Execute single specified command. 

-f file-name

Take input from specified file. 

-h

Display help/usage text. 

-q

Suppress the display of a warning message when a non-root user attempts to use restricted command mode. 

-s cluster-name

Connect to the specified Sun HPC cluster. 

-V

Display mpadmin version information.

-c command - Single Command Option

Use the -c option when you want to execute a single mpadmin command and return upon completion to the shell prompt. For example, the following use of mpadmin -c changes the location of the CRE log file to /home/wmitty/cre_messages:

# mpadmin -c set logfile="/home/wmitty/cre_messages"
#
Note -

Most commands that are available via the interactive interface can be invoked via the -c option. See Chapter 6, mpadmin: Detailed Description for a description of the mpadmin command set and a list of which commands can be used as arguments to the -c option.

-f file-name - Take Input From a File

Use the -f option to supply input to mpadmin from the file specified by the file-name argument. The source file is expected to consist of one or more mpadmin commands, one command per line.

This option can be particularly useful in the following ways:

-h - Display Help

The -h option displays help information about mpadmin.

-q - Suppress Warning Message

Use the -q option to suppress a warning message when a non-root user attempts to invoke a restricted command.

-s cluster-name - Connect to Specified Cluster

Use the -s option to connect to the cluster specified by the cluster-name argument.

-V - Version Display Option

Use the -V option to display the version of mpadmin.

mpadmin Objects, Attributes, and Contexts

mpadmin Objects and Attributes

From the perspective of mpadmin, a Sun HPC cluster consists of a system of objects, which include

Each type of object has a set of attributes whose values can be operated on via mpadmin commands. These attributes control various aspects of their respective objects. For example, a node's enabled attribute can be

mpadmin Contexts

mpadmin commands are organized into four contexts, which correspond to the four types of mpadmin objects. These contexts are summarized below and illustrated in Figure 3-1.

Figure 3-1 mpadmin Contexts

Graphic

mpadmin Prompts

In the interactive mode, the mpadmin prompt contains one or more fields that indicate the current context. Table 3-2 shows the prompt format for each of the possible mpadmin contexts.

Table 3-2 mpadmin Prompt Formats

Prompt Formats 

Context 

[cluster-name]::

Current context = Cluster. 

[cluster-name]Node::

Current context = Node, but not a specific node. 

[cluster-name]N(node-name)::

Current context = a specific node. 

[cluster-name]Partition::

Current context = Partition, but not a specific partition. 

[cluster-name]P(partition-name)::

Current context = a specific partition. 

[cluster-name]N(node-name) Network::

Current context = Network, but not a specific network interface. 

[cluster-name]N(node-name) I(net-if-name)::

Current context = a specific network interface. 

Note -

When the prompt indicates a specific network interface, it uses I as the abbreviation for Network to avoid being confused with the Node abbreviation N.

List Names of Nodes

mpadmin provides various ways to display information about the cluster and many kinds of information that can be displayed. However, the first information you are likely to need is a list of the nodes in your cluster.

Use the list command in the Node context to display this list. In the following example, list is executed on node1 in a four-node cluster.

node1# mpadmin[node0]:: node[node0] Node:: list
    node0
    node1
    node2
    node3
[node0] Node::

The mpadmin command starts up an mpadmin interactive session in the cluster context. This is indicated by the [node0]:: prompt, which contains the cluster name, node0, and no other context information.

Note -

A cluster's name is assigned by the CRE and is always the name of the cluster's master node.

The node command on the example's second line makes Node the current context. The list command displays a list of all the nodes in the cluster.

Once you have this list of nodes, you have the information you need to enable the nodes and to create a partition. However, before moving on to those steps, you might want to try listing information from within the cluster context or the partition context. In either case, you would follow the same general procedure as for listing nodes.

If this is a newly installed cluster and you have not already run the part_initialize script (as described in the previous chapter), the cluster will contain no partitions at this stage. If, however, you did run part_initialize and have thereby created the partition all, you might want to perform the following test.

node1# mpadmin[node0]:: partition[node0] Partition:: list
    all
[node0] Partition::

To see what nodes are in partition all, make all the current context and execute the list command. The following example illustrates this; it begins in the Partition context (where the previous example ended).

[node0]
Partition:: all[node0] P[all]:: list
    node0
    node1
    node2
    node3
[node0] P[all]::

Enabling Nodes

A node must be in the enabled state before MPI jobs can be run on it. To enable a node, make that node the current context and set its enabled attribute. Repeat for each node that you want to be available for running MPI jobs.

The following example illustrates this, using the same four-node cluster used in the previous examples.

node1# mpadmin[node0]:: node0[node0] N[node0]:: set enabled[node0] N[node0]:: node1[node0] N[node1]:: set enabled[node0] N[node1]:: node2[node0] N[node2]:: set enabled[node0] N[node2]:: node3[node0] N[node3]:: set enabled[node0] N[node3]::

Note the use of a shortcut to move directly from the Cluster context to the node0 context without first going to the general Node context. You can explicitly name a particular object as the target context in this way so long as the name of the object is unambiguous--that is, it is not the same as an mpadmin command.

mpadmin accepts multiple commands on the same line. The previous example could be expressed more succinctly as

node1# mpadmin[node0]:: node0 set enabled node1 set enabled node2 set enabled node3 set enabled[node0] N[node3]::

To disable a node, use the unset command in place of the set command.

Creating and Enabling Partitions

You must create at least one partition and enable it before you can run MPI programs on your Sun HPC cluster. Even if your cluster already has the default partition all in its database, you will probably want to create other partitions with different node configurations to handle particular job requirements.

There are three essential steps involved in creating and enabling a partition:

Once a partition is created and enabled, you can run serial or parallel jobs on it. A serial program will run on a single node of the partition. Parallel programs will be distributed to as many nodes of the partition as the CRE determines to be appropriate for the job. Job placement on a partition's nodes is discussed in the Sun MPI 4.0 User's Guide: With CRE.

Example: Creating a Two-Node Partition

The following example creates and enables a two-node partition named part0. It then lists the member nodes to verify the success of the creation.

node1# mpadmin[node0]:: partition[node0] Partition:: create part0[node0] P[part0]:: set nodes=node0 node1[node0] P[part0]:: set enabled[node0] P[part0]:: list
    node0
    node1
[node0] P[part0]::
Note -

There are no restrictions on the number or size of partitions, so long as no node is a member of more than one enabled partition.

Example: Two Partitions Sharing a Node

The next example shows a second partition, part1, being created. One of its nodes, node1, is also a member of part1. 

[node0]
P[part0]:: up[node0] Partition:: create part1[node0] P[part1]:: set nodes=node1 node2 node3[node0] P[part1]:: list
    node1
    node2
    node3
[node0] P[part1]::

Because node1 is shared with part0, which is already enabled, part1 is not being enabled at this time. This illustrates the rule that a node can be a member of more than one partition, but only one of those partitions can be enabled at a time.

If both partitions were enabled at the same time and you tried to run a job on either, the CRE would fail and return an error message. When you want to use part1, you will need to disable part0 first.

Note the use of the up command. The up command moves the context up one level, in this case, from the context of a particular partition (that is, from part0) to the general Partition context.

Shared vs. Dedicated Partitions

The CRE can configure a partition to allow multiple MPI jobs to be running on it concurrently. Such partitions are referred to as shared partitions. The CRE can also configure a partition to permit only one MPI job to run at a time. These are called dedicated partitions.

In the following example, the partition part0 is configured to be a dedicated partition and part1 is configured to allow shared use by up to four processes.

node1# mpadmin[node0]:: part0[node0] P[part0]:: set max_total_procs=1[node0] P[part0]:: part1[node0] P[part1]:: set max_total_procs=4[node0] P[part1]::

The max_total_procs attribute defines how many processes can be active on each node in the partition for which it is being set. In this example, it is set to 1 on part0, which means only one job can be running at a time. It is set to 4 on part1 to allow up to four jobs to be started on that partition.

Note again, that the context-changing shortcut (introduced in "Enabling Nodes") is used in the second and fourth lines of this example.

Customizing Cluster Administration

There are two cluster attributes that you may be interested in modifying, logfile and administrator.

Changing the logfile Attribute

The logfile attribute allows you to log CRE messages in a separate file from all other system messages. For example, if you enter

[node0]:: set logfile=/home/wmitty/cre-messages

CRE will output its messages to the file /home/wmitty/cre-messages. If logfile is not set, CRE messages will be passed to syslog, which will store them with other system messages in /var/adm/messages.

Note -

A full path name must be specified when setting the logfile attribute.

Changing the administrator Attribute

Set the administrator attribute to specify the email address of the system administrator. For example:

[node0]:: set administrator="root@example.com"

Note the use of double quotes.

Quitting mpadmin

Use either the quit or exit command to quit an mpadmin interactive session. Either causes mpadmin to terminate and return you to the shell level. For example:

[node0]:: quitnode1#