IB-HCA ExpressModule Installation

C H A P T E R 2

This chapter describes how to install the Sun Dual Port 4x QDR IB Host Channel Adapter PCIe ExpressModule and verify that it has been installed correctly.

Installing the IB-HCA ExpressModule

Install the ExpressModule

Installing the IB-HCA ExpressModule

The following instructions describe the basic tasks required to install the ExpressModule. Refer to your system installation or service manual for detailed ExpressModule installation instructions.

Install the ExpressModule

1. Halt and power off your system.

2. Install the ExpressModule into the ExpressModule slot, pushing the module’s edge connector into the connector on the chassis.

Ensure that the front plate on the ExpressModule card mounts flush with the chassis panel opening.

3. Connect the 4x end of the InfiniBand I/O cables to the ExpressModule port connectors.

Ensure that the connectors are properly engaged.

Caution - Avoid putting unnecessary stress on the connection. Do not bend or twist the cable near the connectors and avoid cable bends of more than 90 degrees.

4. If not already connected, connect the InfiniBand I/O cables to the appropriate ports on the InfiniBand switch.

5. Turn power back on to the system and allow the server to reboot.

This completes the hardware installation. Proceed to the verification instructions in Verify the Installation With the Solaris 10 OS or Verify the Installation With Linux.

Hot-Swapping With the Linux OS

Use the following procedure to hot-swap the ExpressModule with the Linux OS.

To Hot-Swap the ExpressModule Using the Linux OS

1. Type the following and verify that the IB status is okay:

> ibstat
CA ’mlx4_0’
        CA type: MT26428
        Number of ports: 2
        Firmware version: 2.6.0
        Hardware version: a0
        Node GUID: 0x0002c903000379a4
        System image GUID: 0x0002c903000379a7
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 20
                Base lid: 26
                LMC: 0
                SM lid: 23
                Capability mask: 0x02510868
                Port GUID: 0x0002c903000379a5
        Port 2:
                State: Down
                Physical state: Polling
                Rate: 10
                Base lid: 28
                LMC: 0
                SM lid: 1
                Capability mask: 0x02510868
                Port GUID: 0x0002c903000379a6

2. Load the acpiphp module by typing the following:

Note - Do not use the acpiphp command for hot-plug operations in SuSE Linux SLES10SP2. Use the modprobe pciehp pciehp_poll_mode=1 command instead of the acpiphp command. The remaining steps and commands are the same for the SuSE Linux SLES10SP2 hot-plug procedure.

> modprobe acpiphp

3. If IPoIB is configured, type one of the following:

> ifconfig ib0 down

> ifconfig ib1 down

Where n is the instance number of the ExpressModule.

4. Press the Hot-Swap button on the ExpressModule (see FIGURE 1-3).

Within eight seconds, the green LEDs will go out. The ExpressModule is now physically ready to be removed from the chassis slot.

5. Type the following:

> /etc/init.d/openibd status
HCA driver loaded
 
Configured IPoIB devices:
 
Currently active IPoIB devices:
 
The following OFED modules are loaded:
 
  rdma_ucm
  ib_sdp
  rdma_cm
  ib_addr
  ib_ipoib
  mlx4_core
  mlx4_ib
  ib_mthca
  ib_uverbs
  ib_umad
  ib_ucm
  ib_sa
  ib_cm
  ib_mad
  ib_core

The output displays the driver as loaded but not attached to any device, which indicates the driver is stable.

6. Insert the ExpressModule into chassis slot.

7. After approximately one minute, after all the LEDS are out, press the Hot-Swap button again.

The LEDs start to blink and both the ExpressModule and the ports come up in nine seconds.

8. Type the following to verify that the InfiniBand ports are up:

> ibstat
CA ’mlx4_0’
        CA type: MT26428
        Number of ports: 2
        Firmware version: 2.6.0
        Hardware version: a0
        Node GUID: 0x0002c903000379a4
        System image GUID: 0x0002c903000379a7
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 26
                LMC: 0
                SM lid: 23
                Capability mask: 0x02510868
                Port GUID: 0x0002c903000379a5
        Port 2:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 28
                LMC: 0
                SM lid: 1
                Capability mask: 0x02510868
                Port GUID: 0x0002c903000379a6