Documentation, Support, and Training
Understanding the Installation
Fabric Card Handling Guidelines
Fabric Card and Line Card Fillers
Suggested Tools for Installation
Antistatic Precautions for Installation
Installing the InfiniBand Software Stack
Install the HPC Software Suite
Remove the Shipping Carton Contents
Unsecure the Chassis From the Pallet
Install the Rail Assemblies in the Rack
Install the Chassis in the Rack
Inspecting the Midplane for Installation
Identify the Midplane Fabric Card Connectors
Inspect the Pins on the Rear Side of the Midplane
Identify the Midplane Line Card Connectors
Inspect the Pins on the Front Side of the Midplane
Inspecting the Fabric Cards for Installation
Identify the Fabric Cards for Installation
Inspect the Fabric Card Chassis for Installation
Inspect the Fabric Card Retainer Bolts for Installation
Inspect the Fans for Installation
Inspect the Fabric Card XBOW Connectors for Installation
Inspecting the Line Cards for Installation
Identify the Line Cards for Installation
Inspect the Line Card Chassis for Installation
Inspect the Line Card Retainer Bolts for Installation
Inspect the Line Card XBOW Connectors for Installation
Inspect the Line Card CXP Connectors for Installation
Inspecting the InfiniBand Cables for Installation
Identify the InfiniBand Cable for Installation
Inspect the InfiniBand Cable Hardware for Installation
Inspect the InfiniBand Cable Connectors for Installation
Install the Midplane Stiffener
Install the Fabric Cards or Fabric Card Fillers
Install the Line Cards or Line Card Fillers
Check the Power Supply Status LEDs
Access the CMC From the Serial Management Port
Set the Initial UBoot Parameters
Access the CMC From the Network Management Port
Display the CMC FRU ID Information
Verify the Power Supply Status
Identify the Power Supply Addresses
Display the Power Supply FRU ID Information
Enable the Fabric Cards for Installation
Create the guid.txt File for Installation
Power On the Fabric Cards for Installation
Check the Fabric Card Status LEDs
Identify the Fabric Card Addresses
Display the Fabric Card FRU ID Information
Enable the Line Cards for Installation
Power On the Line Cards for Installation
Check the Line Card Status LEDs
Identify the Line Card Addresses
Display the Line Card FRU ID Information
Attach Cables to the CXP Connectors
Check the CXP Connector Status LEDs
Identify the CXP Connector Addresses
Display the InfiniBand Cable FRU ID Information
Verifying the InfiniBand Fabric
Discover the InfiniBand Fabric Topology
Validate the InfiniBand Fabric and Report Errors
Understanding ILOM on the Switch
The ibdiagnet command performs a collection of tests on the InfiniBand fabric and generates several files which contain parameters and aspects of the InfiniBand fabric.
Identify the prerequisite and subsequent tasks you must perform in conjunction with this procedure, to install the switch.
See the table in Installation Sequence.
On the Linux InfiniBand host, type:
# ibdiagnet
In the following example, the ibdiagnet command checks for 4x link width and 10 Gbyte/sec speed and dumps the Performance Manager counters and then clears them:
# ibdiagnet -lw 4x -ls 10 -pm -pc Loading IBDIAGNET from: /usr/lib64/ibdiagnet1.2 -W- Topology file is not specified. Reports regarding cluster links will use direct routes. Loading IBDM from: /usr/lib64/ibdm1.2 -I- Using port 2 as the local port. -I- Discovering ... 55 nodes (54 Switches & 1 CA-s) discovered. -I--------------------------------------------------- -I- Bad Guids/LIDs Info -I--------------------------------------------------- -I- No bad Guids were found -I--------------------------------------------------- -I- Links With Logical State = INIT -I--------------------------------------------------- -I- No bad Links (with logical state = INIT) were found -I--------------------------------------------------- -I- PM Counters Info -I--------------------------------------------------- -I- No illegal PM counters values were found -I--------------------------------------------------- -I- Links With links width != 4x (as set by -lw option) -I--------------------------------------------------- -I- No unmatched Links (with width != 4x) were found -I--------------------------------------------------- -I- Links With links speed != 10 (as set by -ls option) -I- No unmatched Links (with speed != 10) were found -I--------------------------------------------------- -I- Fabric Partitions Report (see ibdiagnet.pkey for a full hosts list) -I--------------------------------------------------- -I- PKey:0x7fff Hosts:1 full:1 partial:0 -I--------------------------------------------------- -I- IPoIB Subnets Check -I--------------------------------------------------- -I- Subnet: IPv4 PKey:0x7fff QKey:0x00000b1b MTU:2048Byte rate:10Gbps SL:0x00 -W- Suboptimal rate for group. Lowest member rate:40Gbps > group-rate:10Gbps -I--------------------------------------------------- -I- Bad Links Info -I- No bad link were found -I--------------------------------------------------- ---------------------------------------------------------------- -I- Stages Status Report: STAGE Errors Warnings Bad GUIDs/LIDs Check 0 0 Link State Active Check 0 0 Performance Counters Report 0 0 Specific Link Width Check 0 0 Specific Link Speed Check 0 0 Partitions Check 0 0 IPoIB Subnets Check 0 1 Please see /tmp/ibdiagnet.log for complete log ---------------------------------------------------------------- -I- Done. Run time was 42 seconds. #
Note - The Performance Manager counters are available in the ibdiagnet.pm file.
Validate and check errors for the InfiniBand fabric.
Switch Command Reference, ibdiagnet command