C H A P T E R 7 |
Using MCA Parameters With mpirun |
Open MPI uses Modular Component Architecture (MCA) parameters to provide a way to tune your runtime environment. Each parameter corresponds to a specific function. You change the value of the parameter in order to change the function.
Developing an Open MPI application that uses MCA parameters poses a number of advantages. Developers and administrators can customize the Open MPI environment to suit the specific needs of hardware or the operating environment. For example, a system administrator might use MCA parameters to optimize an Open MPI installation on a network so that users only need to run with the default values to obtain the best performance.
This chapter contains the following topics:
In order to understand how MCA parameters fit within Open MPI, you must understand how the Modular Component Architecture is constructed.
The Modular Component Architecture (MCA) is the backbone for much of Open MPI’s functionality. It is a series of frameworks, components, and modules that are assembled at runtime to create an MPI implementation.
An MCA framework manages a specific Open MPI task (such as process launching for ORTE). Each MCA framework supports a single component type, but can support multiple versions of that type. The framework uses the services from the MCA base functionality to find and/or load components.
An MCA component is an implementation of a framework’s interface. It is a standalone collection of code that can be bundled into a plug-in that can be inserted into the Open MPI code base, either at runtime and/or at compile time.
An MCA module is an instance of a component. For example, if a node running an Open MPI application has multiple Ethernet NICs, the Open MPI application will contain one TCP MPI point-to-point component, but two TCP point-to-point modules.
For more information about the Open MPI Modular Component Architecture, see the Open MPI FAQ on runtime tuning at:
http://www.open-mpi.org/faq/?category=tuning
There are three types of frameworks in Open MPI:
You might think of these frameworks as ways to group MCA parameters by function. For example, the OMPI btl framework controls the functions in the byte transfer layer, or BTL (point-to-point byte movement) in the network. All of the MCA parameters that are grouped under btl affect the BTL layer.
In addition to the parameters that are grouped under the individual frameworks, there are top-level MCA parameters that affect the frameworks themselves and specify values to your Open MPI installation.
To view the available top-level parameters in each group, type the following command:
where groupname stands for the parameter group you want to view. For example, to view the available MPI parameters, you would type:
The ompi_info command returns information about your Sun HPC ClusterTools/Open MPI installation. When you issue the command without any modifiers, ompi_info returns the following information:
The ompi_info command can list the parameters for a given component, all the parameters for a specific framework, or all parameters. The ompi_info output for most parameters contains a description of the parameter. The output for any parameter shows the current value of that parameter.
Type the following command at the system prompt:
The output from ompi_info lists all of the installed frameworks, their MCA parameters, and their current values.
Type the following command at the system prompt:
In this example, the command output will list all available MCA parameters for the btl framework.
Type the following command at the system prompt:
In this example, the command output will list all available MCA parameters for the tcp component, which is part of the btl framework.
There are three ways to use MCA parameters with Open MPI:
1. Setting the parameter from the command line using the mpirun --mca command. This method assumes the highest precedence; values set for parameters using this method override any other values specified for the same parameter.
2. Using the parameter as an environment variable. Values for parameters set in this fashion assume the next highest priority.
3. Setting the parameter values in a text file. Parameter values specified using this method have the lowest priority.
To Set MCA Parameters From the Command Line |
Type the following command at the system prompt:
In this example, param-name stands for the name of the MCA parameter you want to set, and value stands for the new value you want to specify for the parameter. For example, the following command sets the value of the mpi_show_handle_leaks parameter to 1 for the specified job:
This sets the value of MCA parameter mpi_show_handle_leaks to 1 before running the program a.out with four processes.
As with other types of environment variables, the syntax for setting MCA parameters as environment variables varies with the type of command shell.
1. Type the following command at the prompt:
where param-name is the name of the MCA parameter you want to set, and value is the desired value for the parameter. For example, the following command sets the mpi_show_handle_leaks parameter to 1:
2. Type the following command:
For example, an export command using the parameter used in the previous step would look like this:
3. Issue the mpirun command with the desired options. For example:
1. Use the setenv command to set the MCA parameter.
where param-name is the name of the MCA parameter you want to set, and value is the desired value for the parameter. The following example shows how to set the mpi_show_handle_leaks parameter to 1.
2. Issue the mpirun command for the program (in this example, a.out).
1. Create a text file, specifying each parameter/value pair on a separate line. Comments are allowed. For example:
2. Name the file mca-params.conf and save it.
You can save the file either to your home directory under $HOME/.openmpi/mca-params.conf, where the parameter values in the file will only affect your jobs, or you can save it to /opt/SUNWhpc/HPC8.2.1c/sun/etc/openmpi-mca-params.conf, where the parameter values in the file affect all users.
The following example shows the output from the ompi_info command for mca_param_files.
The MCA parameter mca_param_files specifies a colon-delimited path of files to search for MCA parameters. Files to the left of the colon have lower precedence; files to the right of the colon have higher precedence. At runtime, mpirun searches the following two files in order when the mca_param_files parameter is set:
1. $HOME/.openmpi/mca-params.conf: The user-supplied set of values takes the highest precedence.
2. $prefix/etc/openmpi-mca-params.conf: The system-supplied set of values has a lower precedence.
In the above example, Open MPI first searches /home/joeuser/.openmpi/mca-params.conf for MCA parameters, and then searches /opt/SUNWhpc/HPC8.2.1c/sun/etc/openmpi-mca-params.conf. If a parameter appears in both locations, the value set in the second file (the file to the right of the colon) is used.
Each MCA framework has a top-level MCA parameter that you can use to select which components are to be used at runtime. In other words, there is an MCA parameter of the same name as each MCA framework (for example, btl) that you can use to include or exclude components from a given run.
You can use top-level parameters in the same way you would use other MCA parameters (for example, you can set them from the command line, as environment variables, or in text files).
For example, the btl MCA parameter is used to control which byte transfer layer (BTL) components are used with mpirun. The value for the btl parameter is a list of components separated by commas, with the optional prefix ^ (caret symbol).
Note - Do not mix “include” instructions with “exclude” instructions in the same command; otherwise, mpirun returns an error. |
Type the following command at the system prompt:
In this example, the components comp1 and comp2 are included for the framework specified by --mca framework. Component comp3 is excluded, since it is preceded by the ^ (caret) symbol.
For example, the following command excludes the tcp and openib components from the BTL framework, and implicitly includes all the other components:
The use of the caret followed by the ellipsis in the command means “Perform the opposite action with the rest of the components.” When the mpirun -- mca command specifies components to be excluded, the caret followed by the ellipsis ellipsis implicitly includes the rest of the components in that framework. When the mpirun --mca command specifically includes components, the caret followed by the ellipsis means “and exclude the components not specified.”
For example, the following command includes only the self, sm, and gm components of btl and implicitly excludes the rest:
The ras_gridengine parameters enable you to specify output from the Open MPI RAS (Resource Allocation Subsystem). The rsh PLM (Process Launch Module) contains the gridengine parameters.
The following example shows the mpirun command line being used to set the MCA parameter plm_gridengine_debug value to 100.
To view a list of the RAS parameters from the command line, use the ompi_info command. The following example shows how to specify the RAS parameters.
There are options available in the MCA PLM and RAS components and modules to allow changes of the default values.
For more information about how to change the values in MCA parameters, see the General Run-time Tuning FAQ on the Open MPi Web site at:
http://www.open-mpi.org/faq/?category=tuning#setting-mca-params
For more information about the Modular Component Architecture and MCA parameters, refer to the following sources:
Copyright © 2010, Oracle and/or its affiliates. All rights reserved.