IB-HCA Card Installation

C H A P T E R 2

The IB-HCA card is a standard PCI Express x8 card with a standard x8 edge connector. Consult the host machine documentation for instructions on how to install a PCI Express card

Installing the IB-HCA Card

Caution - .Electronic components on printed circuit boards are extremely sensitive to static electricity. Ordinary amounts of static electricity generated by your clothing or work environment can damage the electronic equipment. When installing the IB-HCA in a system, use antistatic grounding straps and antistatic mats to help prevent damage due to electrostatic discharge.

To Install the Adapter

Refer to your system installation or service manual for detailed instructions for the following steps:

1. Power off your server, using the standard shutdown procedures described in your system service manual.

Documentation is available on the http://docs.sun.com web site.

2. Remove the cover from the unit to access the card slots and connectors.

3. Select an available PCIe x8 slot and remove the blank front panel.

If you are replacing an existing card, remove the card.

4. Install the IB-HCA card into the slot, pushing the card’s edge connector into the connector on the chassis.

Ensure that the front plate on the IB-HCA card mounts flush with the chassis panel opening.

5. If applicable, install the screw in front plate to secure the IB-HCA card into the chassis.

6. Connect the 4x end of the InfiniBand I/O cable(s) to the IB-HCA port connector(s).

Ensure that the connectors are properly engaged.

Caution - Avoid putting unnecessary stress on the connection. Do not bend or twist the cable near the connectors and avoid cable bends of more than 90 degrees.

7. Replace the cover on the unit.

8. If not already connected, connect the 12x end of the InfiniBand I/O cables to the appropriate ports on the switch or switches.

The IB-HCA ports can be connected to different ports on the same switch or to a port on different switches. See FIGURE 2-1 for an example of cabling an adapter to two switches.

9. Turn power back on to the system and allow the server to reboot.

This completes the hardware installation. Proceed to the verification instructions in To Verify the Installation.

FIGURE 2-1 Adapter Connected to Two Sun IB Switches

Figure shows data ports of the ExpressModule connected to 2 Sun 9-port InfiniBand switches.

To Verify the Installation

The InfiniBand switch should automatically recognize the IB-HCA when it is connected to the fabric if the IB Subnet Manager is running on the switch.

1. Ensure that the cables are connected to the adapter and switches.

2. Verify that the IB Subnet Manager is running on the IB switch.

Refer to the Sun IB Switch 9P Administration Guide (819-0502) manual for information on the IB Subnet Manager.

3. Check that the green LED is illuminated for each port that is connected to the switch.

If the green LED is not on, check the cable connections at the adapter and at the switch.

4. Check that the amber LED is illuminated for each port that is connected to the switch.

If the amber LED is not on, refer to Appendix C, Diagnostic Software and Troubleshooting for more information.

5. Verify that the IB-HCA ports are up and the driver is attached by typing:

# dmesg | grep tavor

The output shows system diagnostic messages that have the string tavor in the message. Included in the output is a message that tells whether the port is up or down. For example:

Feb 17 20:30:42 hostname px_pci: [ID 370704 kern.info] PCI-device:
pciex15b3,6278@0, tavor0
Feb 17 20:30:42 hostname genunix: [ID 936769 kern.info] tavor0 is /pci@7c0/pci@0/pci@8/pciex15b3,6278@0
Feb 17 20:30:42 hostname tavor: [ID 548581 kern.info] tavor0: FW ver:
0004.0007.0190, HW rev: a0
Feb 17 20:30:42 hostname tavor: [ID 827127 kern.info] tavor0: MT23108
Sun Microsystems InfiniBand HCA (0x0003ba0001001c18)
Feb 17 20:30:45 hostname genunix: [ID 979957 kern.info] NOTICE: tavor0:
port 1 up 
Feb 17 20:30:45 hostname genunix: [ID 611667 kern.info] NOTICE: tavor0:
port 2 up
Feb 17 20:30:58 hostname genunix: [ID 408114 kern.info]
/pci@7c0/pci@0/pci@8/pciex15b3,6278@0 (tavor0) online