Documentation, Support, and Training
Understanding the Installation
Fabric Card Handling Guidelines
Fabric Card and Line Card Fillers
Suggested Tools for Installation
Antistatic Precautions for Installation
Installing the InfiniBand Software Stack
Install the HPC Software Suite
Remove the Shipping Carton Contents
Unsecure the Chassis From the Pallet
Install the Rail Assemblies in the Rack
Install the Chassis in the Rack
Inspecting the Midplane for Installation
Identify the Midplane Fabric Card Connectors
Inspect the Pins on the Rear Side of the Midplane
Identify the Midplane Line Card Connectors
Inspect the Pins on the Front Side of the Midplane
Inspecting the Fabric Cards for Installation
Identify the Fabric Cards for Installation
Inspect the Fabric Card Chassis for Installation
Inspect the Fabric Card Retainer Bolts for Installation
Inspect the Fans for Installation
Inspect the Fabric Card XBOW Connectors for Installation
Inspecting the Line Cards for Installation
Identify the Line Cards for Installation
Inspect the Line Card Chassis for Installation
Inspect the Line Card Retainer Bolts for Installation
Inspect the Line Card XBOW Connectors for Installation
Inspect the Line Card CXP Connectors for Installation
Inspecting the InfiniBand Cables for Installation
Identify the InfiniBand Cable for Installation
Inspect the InfiniBand Cable Hardware for Installation
Inspect the InfiniBand Cable Connectors for Installation
Install the Midplane Stiffener
Install the Fabric Cards or Fabric Card Fillers
Install the Line Cards or Line Card Fillers
Check the Power Supply Status LEDs
Access the CMC From the Serial Management Port
Set the Initial UBoot Parameters
Access the CMC From the Network Management Port
Display the CMC FRU ID Information
Verify the Power Supply Status
Identify the Power Supply Addresses
Display the Power Supply FRU ID Information
Enable the Fabric Cards for Installation
Power On the Fabric Cards for Installation
Check the Fabric Card Status LEDs
Identify the Fabric Card Addresses
Display the Fabric Card FRU ID Information
Enable the Line Cards for Installation
Power On the Line Cards for Installation
Check the Line Card Status LEDs
Identify the Line Card Addresses
Display the Line Card FRU ID Information
Attach Cables to the CXP Connectors
Check the CXP Connector Status LEDs
Identify the CXP Connector Addresses
Display the InfiniBand Cable FRU ID Information
Verifying the InfiniBand Fabric
Discover the InfiniBand Fabric Topology
Perform Diagnostics on the InfiniBand Fabric
Validate the InfiniBand Fabric and Report Errors
Understanding ILOM on the Switch
If your InfiniBand fabric uses the Fat-Tree routing algorithm, you must provide a modified listing of fabric card GUIDs, called the root node GUID file, for the Subnet Manager. The /etc/opensm/guid.txt file is such a file. This procedure describes how to create the guid.txt file.
Note - In this procedure, you issue commands to both the CMC and the Linux InfiniBand host.
Identify the prerequisite and subsequent tasks you must perform in conjunction with this procedure, to install the switch.
See the table in Installation Sequence.
Type the following command on the CMC to retrieve the fabric card base GUID:
# getbaseguid fc slot
where slot is the number of the fabric card (0–8). For example:
# getbaseguid fc 0 Base GUID = 0x00bad0fc0009f000 #
Record the GUID to a text editor on your Linux InfiniBand host.
Repeat Step 2 and Step 3 for each fabric card.
Once you have recorded each fabric card’s GUID, you will have a list similar to the following on your Linux InfiniBand host:
0x00bad0fc0009f000 0x00bad0fc0033f100 0x00bad00017fcf200 0x00bad00005fcf300 0x00bad00003fcf400 0x00bad00007fcf500 0x00bad0fc0011f600 0x00bad0b00002f700 0x00bad0b00001f800
Note - If you have fewer than 9 fabric cards, your list will be smaller.
For each GUID, add the following values to make two new sums:
0xa2
0xb2
Note - For GUIDs that end with two zeros (00), you are effectively replacing the zeros with a2 and b2.
For example, 0x00bad0fc0009f000 becomes:
0x00bad0fc0009f0a2
0x00bad0fc0009f0b2
Repopulate your list with the modified GUIDs.
For example:
0x00bad0fc0009f0a2 0x00bad0fc0009f0b2 0x00bad0fc0033f1a2 0x00bad0fc0033f1b2 0x00bad00017fcf2a2 0x00bad00017fcf2b2 0x00bad00005fcf3a2 0x00bad00005fcf3b2 0x00bad00003fcf4a2 0x00bad00003fcf4b2 0x00bad00007fcf5a2 0x00bad00007fcf5b2 0x00bad0fc0011f6a2 0x00bad0fc0011f6b2 0x00bad0b00002f7a2 0x00bad0b00002f7b2 0x00bad0b00001f8a2 0x00bad0b00001f8b2
Save the list on the Linux InfiniBand host as the /etc/opensm/guid.txt file.
Set the permissions for the /etc/opensm/guid.txt file:
# chmod 0644 /etc/opensm/guid.txt #
Restart the Subnet Manager on the Linux InfiniBand host and instruct it to use the guid.txt file.
For example:
# /etc/init.d/opensmd stop Stopping IB Subnet Manager..-. [ OK ] # /usr/sbin/opensm -R ftree -A -a /etc/opensm/guid.txt -s 120 -B
Power on the fabric cards.
Switch Command Reference, getbaseguid command
Switch Command Reference, opensm command