Internet Protocol Over InfiniBand

This chapter describes configuration aspects of running the Internet Protocol over InfiniBand (IPoIB) and contains the following section:

4.1 Configuring IPoIB on Linux

You might decide to change your IPoIB configuration for a variety of reasons, including the installation of an additional IB NEM.

4.1.1 Install IPoIB Driver

1. Detemine whether the IPoIB driver is already installed by entering the
lsmod | grep ib command.

In the following sample output, note that the driver, ib_ipoib, is not listed.

> lsmod | grep ib
ib_sdp       45340 0
rdma_cm      26760 1 ib_sdp
ib_addr      10504 1 rdma_cm
ib_local_sa  14232 1 rdma_cm
findex       6528 1 ib_local_sa
ib_ipath     70552 0
ipath_core   179652 1 ib_ipath
ib_mthca     139184 0
ib_uverbs    47536 0
ib_umad      19888 0
ib_ucm       21512 0
ib_sa        18196 2 rdma_cm,ib_local_sa
ib_cm        39952 2 rdma_cm,ib_ucm
ib_mad       43176 5 ib_local_sa,ib_mthca,ib_umad,ib_sa,ib_cm
ib_core      59520 11
ib_sdp,rdma_cm,ib_local_sa,ib_ipath,ib_mthca,ib_uverbs,ib_umad, ib_ucm,ib_sa,ib_cm,ib_mad

> lsmod | grep ib
ib_ipoib     59800 0
ib_sdp       45340 0
rdma_cm      26760 1 ib_sdp
ib_addr      10504 1 rdma_cm
ib_local_sa  14232 1 rdma_cm
.
.
.
ib_core      59520 11
ib_sdp,rdma_cm,ib_local_sa,ib_ipath,ib_mthca,ib_uverbs,ib_umad, ib_ucm,ib_sa,ib_cm,ib_mad

> ifconfig ib0
ib0     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
        BROADCAST MULTICAST  MTU:2044  Metric:1
        RX packets:0 errors:0 dropped:0 overruns:0 frame:0
        TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:128
        RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

4.1.2 Change IPoIB Configuration Without Rebooting

1. Start the InfiniBand IP network by entering the ifconfig command and assigning a valid IP address for ib0.

> ifconfig ib0 10.0.0.50/24
ib0     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
        inet addr:10.0.0.50  Bcast: 10.0.0.255  Mask:255.255.255.0
        UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
        RX packets:0 errors:0 dropped:0 overruns:0 frame:0
        TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:128
        RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

If ib0 were unconfigured, it would appear without an IP address as shown in the following output.

> ifconfig ib0
ib0     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
        BROADCAST MULTICAST  MTU:2044  Metric:1 
        RX packets:0 errors:0 dropped:0 overruns:0 frame:0
        TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:128
        RX bytes:0 (0.0 b)  TX bytes:0 ( 0.0 b)

> route
Kernel IP routing table
Destination Gateway    Genmask        Flags  Metric Ref  Use  Iface
10.0.0.0    *          255.255.255.0  U      0      0    0    ib0
10.8.134.0  *          255.255.255.0         0      0    0    eth0
169.254.0.0 *          255.255.0.0    U      0      0    0    eth0
default     ban3rtr0d0 0.0.0.0        UG     0      0    0    eth0

3. As shown in the following example, you can enter the ping command to see another IPoIB node on the 10.0.0 subnet:

> ping 10.0.0.79
PING 10.0.0.79 (10.0.0.79) 56(84) bytes of data.
64 bytes from 10.0.0.79: icmp_seq=0 ttl=255 time=1.82 ms
64 bytes from 10.0.0.79: icmp_seq=1 ttl=255 time=0.082 ms
64 bytes from 10.0.0.79: icmp_seq=2 ttl=255 time= 0.062 ms
 
--- 10.0.0.79 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.062/0.655/1.823/0.825 ms, pipe 2

At this point, the IPoIB network is active and properly configured without rebooting.

4.1.3 Change IB Startup Behavior on Linux

To change the InfiniBand startup behavior, edit the openib.conf configuration file (full path: /etc/infiniband/openib.conf).

In the following example, openib.conf specifies that whenever the system boots, the InfiniBand services, IPoIB, and the SDP IP service are to start up automatically (ONBOOT=yes, IPOIB_LOAD=yes, SDP_LOAD=yes). However, openib.conf specifies that the SRP service is not to start up automatically (SRP_LOAD=no). You can alter any and all of these parameters.

# Start HCA driver upon boot
ONBOOT=yes
# Load UCM module
UCM_LOAD=no
# Load RDMA_CM module
RDMA_CM_LOAD=no
# Load RDMA_UCM module
RDMA_UCM_LOAD=no
# Load MTHCA
MTHCA_LOAD=yes
# Load IPATH
IPATH_LOAD=yes
# Load IPoIB
IPOIB_LOAD=yes
# Load SDP module
SDP_LOAD=yes
# Load SRP module
SRP_LOAD=no
# Load RDS module
RDS_LOAD=no

2. Create or edit the ifcfg-ibn file to configure an individual network interface.

For each InfiniBand network interface, you need a corresponding /etc/sysconfig/network-scripts/ startup file (ifcfg-ib0). As an example, the startup file for ib0, might look something like the following.

more /etc/sysconfig/network-scripts/ifcfg-ib0
 DEVICE=ib0
 BOOTPROTO=static
 IPADDR=10.0.0.50
 NETMASK=255.255.255.0
 NETWORK=10.0.0.0
 BROADCAST=10.0.0.255
 ONBOOT=yes

The ONBOOT=yes parameter indicates that the corresponding IP network interface is to automatically start up when the system boots. Specifying ONBOOT=no will "configure" the interface but not start it.

Generally, if you have enabled IPoIB services, the IB stack installation scripts automatically create the ifcfg-ibn configuration files for all IB network interfaces present. If you install a NEM after you have installed the IB stack, you need to manually create the ifcfg-ibn files for the newly installed network interfaces.