C H A P T E R 3 - Configuring IP Over InfiniBand

C H A P T E R 3

Configuring IP Over InfiniBand

This chapter contains the following sections:

IPoIB Configuration

Managing IB Devices for IPoIB

IPoIB Configuration

IP Over InfiniBand (IPoIB) is automatically configured when the system is booted. The following procedure can be used when making changes to your configuration. See System Administration Guide: Basic Administration (817-1985) for Solaris setup information.

Configuring IPoIB

1. Confirm that the IB-HCA ports are up.

To do this, type the command:

# dmesg | grep tavor

The output shows recently printed diagnostic messages that have the string "tavor" in the message. Included in the output is a message that tells whether the port is up or down.

For example:

Dec 10 16:41:05 hostname pcplusmp: [ID 637496 kern.info] pcplusmp: pci15b3,5a44 (tavor) instance 0 vector 0x1c ioapic 0x4 intin 0x0 is bound to cpu 1

Dec 10 16:41:05 hostname pci_pci: [ID 370704 kern.info] PCI-device: pci15b3,5a44@0, tavor0

Dec 10 16:41:05 hostname genunix: [ID 936769 kern.info] tavor0 is /pci@0,0/pci1022,7450@b/pci15b3,5a46@1/pci15b3,5a44@0

Dec 10 16:41:05 hostname tavor: [ID 548581 kern.info] tavor0: FW ver:

0003.0002.0000, HW rev: a1

Dec 10 16:41:05 hostname tavor: [ID 827127 kern.info] tavor0: MT23108

InfiniHost Mellanox Technologies (0x0002c9010976bb40)

Dec 10 16:41:06 hostname tavor: [ID 979957 kern.info] NOTICE: tavor0:

port 1 up (link width: 4x)

Refer to dmesg(1M) for more information.

2. Create the InfiniBand devices using the following command:

# devfsadm -C

A /dev/ibdn device is created for each IB-HCA port in the system where n is the port number. Refer to devfsadm(1M) for more information.

3. Verify InfiniBand devices.

Check that the InfiniBand devices were created. To do this, type the following command:

# ls -l /dev/ibd*

A list of InfiniBand devices (/dev/ibd*) is displayed.

4. Plumb each of the IB devices.

To do this, type the following command for each IB-HCA port in the system:

# ifconfig ibdn plumb host-ipv4-address up

where n is the instance number of the interface and host-ipv4-address is the designated IPv4 address for the device.

If you are using IPv6, use the following command instead:

# ifconfig ibdn inet6 plumb host-ipv6-address up

5. Verify that the devices are plumbed.

To do this, type the command:

# ifconfig -a

The output includes information on the IB devices.

6. Check that data can be send over the InfiniBand link.

Use the ping command to send ICMP (ICMP6) ECHO_REQUEST packets and receive ICMP (ICMP6) ECHO_RESPONSE packets between InfiniBand hosts. For example:

# ping host-ip-address

where host-ip-address is the IP address of another IB host. See ping(1M) for details.

Repeat this procedure on all hosts that are connected to the IB switch.

Managing IB Devices for IPoIB

For information on managing the IB devices, see the System Administration Guide: Devices and Files System (817-5093).