10 Configuring Ethernet over InfiniBand

This chapter describes how to set up Ethernet over InfiniBand (EoIB) network configuration for Oracle Big Data Appliance.

This chapter discusses the following sections:

Introduction to Virtual NIC (VNIC)
Setting Up Ethernet over InfiniBand (EoIB)

10.1 Introduction to Virtual NIC (VNIC)

A virtual network interface card (VNIC) maps an Ethernet connector on the Sun Network QDR InfiniBand Gateway Switch to a network interface within the server. A connector (0A-ETH-1 to 0A-ETH-4, and 1A-ETH-1 to 1A-ETH-4) hosts a 10 GbE port. On each Sun Network QDR InfiniBand Gateway Switch on Oracle Big Data Appliance, you can have a maximum of eight 10 GbE uplinks. In most scenarios, the number of 10 GbE uplinks is less than eight.

The network interface on the server, which is produced by creating a VNIC on the Sun Network QDR InfiniBand Gateway Switch, uses the ethx naming convention, such as eth8 and eth9. You can assign multiple VNICs to a server. In that case, they can be different connectors on the same Sun Network QDR InfiniBand Gateway Switch or on different ones. The default configuration maps one connector on each of the two switches, and bonds or balances the pair for increased availability.

In Oracle Big Data Appliance, you manually create VNICs in Sun Network QDR InfiniBand Gateway Switch.

10.2 Setting Up Ethernet over InfiniBand (EoIB)

To set up Ethernet over InfiniBand connectivity:

Use an SSH client, such as PuTTY, to log in to a Sun Network QDR InfiniBand Gateway Switch as root. For example, log in to bda1sw-ib2 as root.

At the command prompt, run the following command:

# listlinkup | grep Bridge

This is an example of the output of this command:

Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled)
Bridge-0 Port 0A-ETH-2 (Bridge-0-2) down (Enabled)
Bridge-0 Port 0A-ETH-3 (Bridge-0-1) down (Enabled)
Bridge-0 Port 0A-ETH-4 (Bridge-0-1) down (Enabled)
Bridge-1 Port 1A-ETH-1 (Bridge-1-2) down (Enabled)
Bridge-1 Port 1A-ETH-2 (Bridge-1-2) down (Enabled)
Bridge-1 Port 1A-ETH-3 (Bridge-1-1) down (Enabled)
Bridge-1 Port 1A-ETH-4 (Bridge-1-1) down (Enabled)

In this example, the uplink is 0A-ETH-1; all others are down.

Determine the globally unique identifiers (GUIDs) of an Oracle Big Data Appliance server as follows:
1. On the server that requires the VNIC, log in as root, and run the ibstat command on the command line. For example, log in to bda1node01 as root. This command displays port information, as shown in the following example output:
```
# ibstat
CA 'mlx4_0'
        CA type: MT26428
        Number of ports: 2
        Firmware version: 2.9.1000
        Hardware version: b0
        Node GUID: 0x0021280001cf4b56
        System image GUID: 0x0021280001cf4b59
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 137
                LMC: 0
                SM lid: 147
                Capability mask: 0x02510868
                Port GUID: 0x0021280001cf4b57
                Link layer: IB
        Port 2:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 138
                LMC: 0
                SM lid: 147
                Capability mask: 0x02510868
                Port GUID: 0x0021280001cf4b58
                Link layer: IB
```
  In the output, information for two ports is displayed. From this example, you must determine which port GUID to use. Before doing so, write down the two Base lid values shown in the output. In the example output, Port 1 has a Base lid value of 137, and Port 2 has a Base lid value of 138.
2. On the same server, run the following command on the command line to report information about all active links in the InfiniBand fabric:
```
# iblinkinfo.pl -R | grep hostname
```
  In this command, hostname is the name of the server, such as bda1node01. You can also specify the bonded IP over InfiniBand (IPoIB) address of the server.
  
  The following is the example output of this command:
```
148  6[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>  138  2[  ] "bda1node01 HCA-1" ( )
149  6[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>  137  1[  ] "bda1node01 HCA-1" ( )
```
  From this example output, note the switch lid values. Port 2 base lid of 138 is associated with switch lid 148 (the first column in the output). Port 1 base lid of 137 is associated with switch lid 149.
Determine which gateway switch is associated with the switch lids (148 and 149) by comparing the first column of the iblinkinfo output to the lid value of the ibswitches command as follows:
1. On the server, issue the ibswitches command. The example output of this command follows.
```
# ibswitches
Switch  : 0x002128df0f0ac0a0 ports 36 "SUN IB QDR GW switch bda1sw-ib3 10.133.43.37" enhanced port 0 lid 148 lmc 0
Switch  : 0x0021284690eea0a0 ports 36 "SUN DCS 36P QDR bda1sw-ib1 10.133.43.35" enhanced port 0 lid 147 lmc 0
Switch  : 0x002128df348ac0a0 ports 36 "SUN IB QDR GW switch bda1sw-ib2 10.133.43.36" enhanced port 0 lid 149 lmc 0
```
2. In this example output, identify the switches that lid values 148 and 149 are associated with.
  
  Lid 148 is associated with gateway switch bda1sw-ib3 and GUID 0x002128df0f0ac0a0.
  
  Lid 149 is associated with gateway switch bda1sw-ib2 and GUID 0x002128df348ac0a0. The following examples use this switch.
Define a dummy media access control (MAC) address in the following format:

last three octets from bda1sw-ib2 switch ib GUID : last three octets of the administrative IP of the server in hexadecimal

For example, the bda1sw-ib2 switch GUID is 00:21:28:df:34:8a:c0:a0 (0x002128df348ac0a0). The last three octets are 8a:c0:a0.

Assume that the administrative IP of the server that requires the VNIC is 192.168.1.1. The last three octets of the IP address are 168.1.1. Convert each decimal to hexadecimal, and it is a8:1:1.

Then you can define the MAC address as follows:

8a:c0:a0:a8.1.1

Note:
Ensure that the MAC address is unique. Only even numbers are supported for the most significant byte of the MAC address (unicast). The address used here is an example only.
As root, log in to the switch identified in Step 4. Use its IP address or host name to log in. In this example, the switch is bda1sw-ib2.
Issue a command like the following to create a VNIC, using the appropriate GUID and MAC address for your system:

# createvnic 0A-ETH-1 -guid 00:21:28:df:34:8a:c0:a0 -mac 8a:c0:a0:a8.1.1 -pkey default

This command does not associate the VNIC with a VLAN. See Chapter 11 for information about creating VLANs.

Oracle Big Data Appliance uses a single partition (the default partition).

To verify the VNIC, run the showvnics command. The following example output is displayed:

# showvnics
ID  STATE FLG IOA_GUID         NODE          IID  MAC               VLN PKEY   GW
--- ----- --- ---------------- ------------ ----- ----------------- ---- ---- ---- 
228 UP      N 0021280001CF4B57 bda1node01 0000 CE:4B:57:85:2A:FC NO ffff 0A-ETH-1

On the server, issue this command to display the list of VNICs available on the compute node:
```
# mlx4_vnic_info --list
eth8
```
Repeat Steps 7 to 9 to create another VNIC on the same server. The second VNIC appears on the server as eth9.

Creating a Bonded Interface

You must configure these two Ethernet over InfiniBand (EoIB) interfaces as a bonded interface named bondeth0. This procedure requires that you manually create three files:

ifcfg-eth8: Describes the eth8 connection
ifcfg-eth9: Describes the eth9 connection
ifcfg-bondeth0: Bonds the two connections in active-passive failover mode

To create a bonded interface from the EoIB interfaces:

Log in as root to the same server where you created the VNICs and change to the /etc/sysconfig/network-scripts directory.
Using a text editor such as vi, create or edit a file named ifcfg-eth8 with the following contents. Replace hostname with the name of the server, and address with the dummy MAC address created in Step 5 of the previous procedure.
```
#ifcfg_eth8 for host hostname
DEVICE=eth8
MASTER=bondeth0
SLAVE=yes
ONBOOT=yes
MTU=65520
BOOTPROTO=none
HOTPLUG=no
IPV6INIT=no
HWADDR=address
```
Create or edit a second file named ifcfg-eth9 for eth9, using its dummy MAC address.

Create or edit a file named ifcfg-bondeth0 with the following parameters. Verify that the settings are correct for your network.

#ifcfg-bondeth0 for server bda1node01 generated by BDAeiob.g
DEVICE=bondeth0
TYPE=ETHERNET
BROADCAST=203.0.115.255
ONBOOT=yes
#MTU=65520
BOOTPROTO=none
NETMASK=255.255.255.0
IPV6INIT=no
NETWORK=203.0.114.0
IPADDR=203.0.113.10
BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000"

Start the new bondeth0 interface, and restart the server for the changes to take effect.