| Sun Blade 6048 Switched InfiniBand Network ExpressModule User’s Guide |    
|
| Sun Blade 6048 Switched InfiniBand Network ExpressModule User’s Guide |
|
| C H A P T E R 3 |
|
InfiniBand Software Overview |
InfiniBand is a network architecture that is designed for the large-scale interconnection of computing and I/O nodes through a high-speed switched fabric. To operate InfiniBand on a Sun Blade 6048 Series Modular System, you need an InfiniBand HCA (provided by the IB NEM) and an InfiniBand software stack.
This chapter provides an overview and installation instructions for the InfiniBand software stack for the Linux and Windows operating systems.
Consult the Sun Blade 6048 Series Product Notes for the most recent information about supported operating systems, firmware and software updates, and other issues not covered in the main product documentation.
This chapter contains the following sections:
If you have installed CentOS 4.4 (Final), RHEL4 u4, RHEL4 u5 or RHEL5 u1on a Sun Blade Server Module and you have installed the bundled drivers, you do not need to install or configure additional drivers to support the IB NEM. The bundled drivers support basic IB NEM operation.
Specifically, CentOS 4.4 contains support in the kernel for HCA hardware produced by Mellanox (mthca driver). The kernel also includes core InfiniBand modules, which provide the interface between the lower-level hardware driver and the upper-layer InfiniBand protocol drivers and provide user space access to InfiniBand hardware.
The kernel also includes the Sockets Direct Protocol (SDP) driver, IP over Infiniband (IPoIB), and the SCSI RDMA Protocol (SRP) driver.
CentOS 4.4 includes the following user space packages:
The packages selected to support any given configuration will vary. TABLE 3-1 lists the packages considered the absolute minimum needed to support the environment described in this guide.
If you elected not to install these packages when installing CentOS 4.4, or if you want to upgrade your drivers, you can install these packages at any time from the OS distribution source or by downloading the required files from OpenFabrics.org. For information on both of these procedures, see Section 3.1.2, Installing the InfiniBand Drivers on Linux.
As the popularity of InfiniBand technology increases, the number of Linux distributions and open source organizations producing drivers and tools will increase. For up-to-date information, check with open source organizations and your current vendors.
The OpenFabrics organization is the Open Software solution in the InfiniBand software space and OpenFabrics Enterprise Distribution (OFED) is the InfiniBand suite of software produced by this organization. Various vendors contribute their drivers (and other software components) to OFED.
TABLE 3-2 lists the tested Linux platforms and the corresponding OFED release.
OFED contains the following components:
If you did not install the InfiniBand drivers when installing the Linux OS, you can install them at any time from the OS distribution source or by downloading the necessary files from OpenFabrics.org.
To do so, choose one of the following procedures:
If you need to determine whether or not the drivers are already installed, see Section 3.1.2.3, Verify Driver Installation on Linux.
To install the InfiniBand drivers, you need access to the Red Hat Package Manager (RPM) files. Access to these files is dependent on your individual installation configuration (net boot, CD/DVD boot, .iso files, and so on). When you decide on the appropriate access method and package selection, you can add the packages to the KickStart configuration file for automatic inclusion in future installations.
| Note - All packages have a .i386.rpm extension (as shown in the following procedure). On a 64-bit system, all packages have a .x86_64.rpm extension instead. |
1. Enter the rpm -ivh command for each InfiniBand package that you need to install.
Packages must be installed in the following order:
The following example shows the installation of one package (libibcommon) and the resulting dialog:
2. If you are running the CSH or TCSH shell, enter the rehash command to rebuild the shell’s view of available executables.
3. Enter the ibstat command to verify that the OS sees the IB NEM.
4. (Optional) You can enter the ibnetdiscover command to verify the presence of an operational IB fabric.
For an example of the output of this command, see Section 3.1.2.3, Verify Driver Installation on Linux.
5. (Optional) You can check the status of the ib0 network interface to determine whether the ib_ipoib driver is installed.
For details on this step, see Section 4.1.1, Install IPoIB Driver.
1. On the Sun Blade Server Module, log in as root and copy the required files (OFED1.2.5.tgz in this example) from the following location:
http://www.openfabrics.org/downloads.html
| Note - You need write access to the files to execute the install script. |
2. From root, extract the files by entering the following command:
3. From the OFED-1.2.5 directory, initiate the installation process by entering the following command:
4. When the InfiniBand OFED Distribution Software Installation menu appears, enter option 2 (Install OFED Software).
5. When the Select OFED Software menu appears, enter option 3 (All packages).
6. When you are asked if you wish to create/install an MPI RPM with gcc,
enter Y.
The following compilers on your system can be used to build/install MPI: gcc Do you wish to create/install an MPI RPM with gcc? [Y/n]: |
7. When you are asked if you wish to create/install an openmpi RPM with gcc, enter Y.
The following compilers on your system can be used to build/install openmpi: gcc Do you wish to create/install an openmpi RPM with gcc? [Y/n]: |
The installation script then lists the OFED packages that it will build. See the following sample output.
You are prompted to configure InfiniBand IP support.
9. Enter Y when asked if you want to include IPoIB configuration files.
Do you want to include IPoIB configuration files (ifcfg-ib*)? [Y/n]:
10. Press Enter to accept the default when prompted to enter a temporary directory for OFED.
RPM build process requires a temporary directory. Please enter the temporary directory [/var/tmp/OFED]: |
11. Press Enter to accept the default when prompted for the OFED installation directory.
Please enter the OFED installation directory [/usr/local/ofed]:
At this point, the installer begins compiling InfiniBand packages. The process of building packages takes approximately 15-20 minutes.
The system displays output like the following:
Installation then begins. You see the following message.
Removing previous InfiniBand Software installation Running /bin/rpm -e libibverbs libibverbs-devel libibverbs-utils... |
The actual installation takes about one minute.
Assuming the IB NEM hardware is installed (and, therefore, an InfiniBand HCA is present), you are prompted to configure InfiniBand IP support.
12. Enter Y in response to the following prompt:
Do you want to configure IPoIB interfaces [Y/n]?
The default IPoIB interface configuration is based on DHCP. A special patch for DHCP is required for supporting IPoIB. The patch is available under:
OFED-1.0/docs/dhcp
If you do not have DHCP, you must change this configuration in the following steps.
The system next displays the current configuration.
13. When asked if you want to change the configuration as displayed, enter y.
The current IPOIB configuration for ib0 is: DEVICE=ib0 BOOTPROTO=dhcp ONBOOT=yes Do you want to change this configuration? [y/N]: |
The configuration script guides you through the changes one at a time. See the following as an example.
14. Enter Y to save the configuration.
If you have entered a valid IP configuration for ib0, you are now properly configured for IPoIB operations.
15. Iterate the InfiniBand configuration over all InfiniBand interfaces.
You must enter a valid IP configuration for each network interface.
Once all IPoIB interfaces have been configured, you are prompted as follows to configure OpenSM for the blade.
Do you want to configure OpenSM [Y/n]?
16. Enter n to complete this part of the installation.
You should see a message like the following:
For information on OpenSM, see Section 3.2, Running OpenSM.
The InfiniBand OFED Distribution Software Installation Menu is displayed.
The Sun Blade Server Module is configured now to start up the InfiniBand software on reboot (ONBOOT=yes).
If this is not the desired behavior, you can edit the /etc/infiniband/openib.conf file, changing ONBOOT to equal no. You can also manually control basic InfiniBand behavior by entering the following command:
/etc/init.d/openibd { start | stop | status }
After successful installation, reboot the Server Module. After reboot, the Server Module should appear as a functional member of the InfiniBand fabric.
1. Verify that the Linux software driver is installed and attached to the IB NEM by entering the openibd status command.
| Note - When using the openibd command, enter the entire path as shown in the example. |
The following example shows the IB driver installed, running, and presenting one IB HCA channel or network device (ibn) to the OS. In the example, the Linux network device appears as ib0.
> /etc/init.d/openibd status HCA driver loaded Configured devices: ib0 Currently active devices: ib0 The following modules are also loaded: ib_cm ip_ipoib . . . |
2. To view details of operational status, enter the ibstat command.
The following example shows one operational IB port into the IB fabric (or network). The LinkUp state indicates active participation in an IB fabric. The port is present as lid 69 and it is being managed by lid 2.
You can also verify that the InfiniBand fabric is operational by entering the ibnetdiscover command. The output from this command lists all the nodes, as shown in the following sample output.
| Note - The output from ibnetdiscover can be lengthy in a large IB fabric. |
In the InfiniBand architecture, a subnet manager (SM) is required for the InfiniBand fabric to function properly. The SM discovers all the nodes on the fabric and assigns the local identifiers (LIDs) in the HCAs. The subnet manager also sets up the routing tables in the switches to support routing packets between nodes.
To meet these needs, OFED and the CentOS supply OpenSM, an open source subnet manager. OpenSM can initialize and configure the subnet as well as keep the subnet operational when the network topology and nodes change. OpenSM runs as a system daemon on at least one of the host machines in the InfiniBand fabric.
The OpenSM application also contains the subnet administrator (SA), an associated component that acts like a database and can be affected by end node requests. OpenSM supports querying as well as event forwarding. Applications send queries to the SA to discover the path records for remote nodes, which are needed to establish connections between endpoints on the fabric.
| Note - Two instances of OpenSM running concurrently on the same port will result in a system crash. |
For more information on OpenSM, see the README file for OFED on CentOS.
| Sun Blade 6048 Switched InfiniBand Network ExpressModule User’s Guide | 820-2189-10 |
|
Copyright © 2008, Sun Microsystems, Inc. All Rights Reserved.