C H A P T E R 3 - Configuring IP Over InfiniBand

C H A P T E R 3

Configuring IP Over InfiniBand

This chapter describes how to configure IP over InfiniBand (IPoIB). This chapter contains the following sections:

IPoIB Configuration

IPoIB Configuration

The following procedure can be used when making changes to your configuration. See System Administration Guide: Basic Administration (817-1985) for Solaris setup information.

Configuring IPoIB

1. Confirm that the IB-EM ports are up.

To do this, type the command:

# dmesg | grep tavor

The output shows recently printed diagnostic messages that have the string "tavor" in the message. Included in the output is a message that tells whether the port is up or down.

For example:

Feb 17 20:30:42 hostname px_pci: [ID 370704 kern.info] PCI-device: pciex15b3,6278@0, tavor0

Feb 17 20:30:42 hostname genunix: [ID 936769 kern.info] tavor0 is /pci@7c0/pci@0/pci@8/pciex15b3,6278@0

Feb 17 20:30:42 hostname tavor: [ID 548581 kern.info] tavor0: FW ver: 0004.0007.0190, HW rev: a0

Feb 17 20:30:42 hostname tavor: [ID 827127 kern.info] tavor0: MT23108 Sun Microsystems InfiniBand HCA (0x0003ba0001001c18)

Feb 17 20:30:45 hostname genunix: [ID 979957 kern.info] NOTICE: tavor0: port 1 up

Feb 17 20:30:45 hostname genunix: [ID 611667 kern.info] NOTICE: tavor0: port 2 up

Feb 17 20:30:58 hostname genunix: [ID 408114 kern.info] /pci@7c0/pci@0/pci@8/pciex15b3,6278@0 (tavor0) online

Refer to the Man Page dmesg(1M) for more information.

2. Create the InfiniBand devices using the following command:

# devfsadm -C

A /dev/ibdn device is created for each IB-EM port in the system where n is the port number. Refer to devfsadm(1M) for more information.

3. Verify InfiniBand devices.

Check that the InfiniBand devices were created. To do this, type the following command:

# ls -l /dev/ibd*

A list of InfiniBand devices (/dev/ibd*) is displayed.

# ls -l /dev/ibd*

lrwxrwxrwx    1 root     other        29 Apr 4 15:07 /dev/ibd -> ../devices/pseudo/clone@0:ibd

lrwxrwxrwx    1 root     root         88 Apr 4 15:26 /dev/ibd1 -> ../devices/pci@1b,0/pci1022,7450@3/pci15b3,5a46@1/pci15b3,5a44@0/ibport@1,8001 ,ipib:ibd1

lrwxrwxrwx    1 root root             88 Apr 4 15:26 /dev/ibd3 -> ../devices/pci@1b,0/pci1022,7450@3/pci15b3,5a46@1/pci15b3,5a44@0/ibport@2,8001 ,ipib:ibd3

4. Plumb each of the IB devices.

To do this, type the following command for each IB-EM port in the system:

# ifconfig ibdn plumb host-ipv4-address up

where n is the instance number of the interface and host-ipv4-address is the designated IPv4 address for the device.

If you are using IPv6, use the following command instead:

# ifconfig ibdn inet6 plumb host-ipv6-address up

5. Verify that the devices are plumbed.

To do this, type the command:

# ifconfig -a

The output includes information on the IB devices.

# ifconfig -a

lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232

index 1

        inet 127.0.0.1 netmask ff000000

ibd1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 2044 index 2

        inet 192.128.91.198 netmask ffffff00 broadcast 192.128.91.255

        ipib 0:0:4:7:0:0:0:0:0:0:0:1:0:3:ba:0:1:0:1c:19

ibd3: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 2044 index 3

        inet 192.128.92.198 netmask ffffff00 broadcast 192.128.92.255

        ipib 0:0:4:7:0:0:0:0:0:0:0:1:0:2:c9:2:0:0:55:d9

6. Check that data can be send over the InfiniBand link.

Use the ping command to send ICMP (ICMP6) ECHO_REQUEST packets and receive ICMP (ICMP6) ECHO_RESPONSE packets between InfiniBand hosts. For example:

# ping host-ip-address

where host-ip-address is the IP address of another IB host. See ping(1M) for details.

Repeat this procedure on all hosts that are connected to the IB switch.