To set up Ethernet over InfiniBand connectivity for an Exalogic compute node running Oracle Linux, complete the following steps:
Use an SSH client, such as PuTTY, to log in to a Sun Network QDR InfiniBand Gateway Switch. Oracle recommends that you log in as the root user. For example, log in to el01gw04 as root.
At the command prompt, run the following command:
el01gw04# listlinkup | grep Bridge
The following is an example of the output of the listlinkup command:
Connector 0A-ETH Present Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled) Bridge-0 Port 0A-ETH-2 (Bridge-0-2) up (Enabled) Bridge-0 Port 0A-ETH-3 (Bridge-0-1) up (Enabled) Bridge-0 Port 0A-ETH-4 (Bridge-0-1) up (Enabled) Bridge-0 Port 1A-ETH-1 (Bridge-1-2) down (Enabled) Bridge-0 Port 1A-ETH-2 (Bridge-1-2) down (Enabled) Bridge-0 Port 1A-ETH-3 (Bridge-1-1) up (Enabled) Bridge-0 Port 1A-ETH-4 (Bridge-1-1) up (Enabled)
From this output, identify the uplinks. You can determine that you can use any of the following Ethernet connectors for creating a VNIC:
0A-ETH-1
0A-ETH-2
0A-ETH-3
0A-ETH-4
1A-ETH-3
1A-ETH-4
Note:
This procedure uses 1A-ETH-3 as an example.
Determine GUIDs of the Exalogic compute node that requires the VNIC, as follows:
On the compute node that requires the VNIC, log in as root, and run the ibstat command on the command line. For example, log in to el01cn01 as root.
Example:
el01cn01# ibstat
CA 'mlx4_0'
CA type: MT26428
Number of ports: 2
Firmware version: 2.7.8100
Hardware version: b0
Node GUID: 0x0021280001a0a364
System image GUID: 0x0021280001a0a367
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 120
LMC: 0
SM lid: 6
Capability mask: 0x02510868
Port GUID: 0x0021280001a0a365
Link layer: IB
Port 2:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 121
LMC: 0
SM lid: 6
Capability mask: 0x02510868
Port GUID: 0x0021280001a0a366
Link layer: IB
In the output, information about two ports is displayed. Identify the GUID and Base lid of the port that you want to use for creating the VNIC.
For the example illustrated in this procedure, we will use the port with GUID 0x0021280001a0a366 and Base lid 121.
On the same compute node, run the following command to view information about all the active links in the InfiniBand fabric:
hostname# iblinkinfo.pl -R | grep hostname
hostname is the name of the compute node. You can also specify the bonded IPoIB address of the compute node.
Example:
el01cn01# iblinkinfo.pl -R | grep el01cn01 65 15[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 121 2[ ] "el01cn01 EL-C 192.168.10.29 HCA-1" (Could be 5.0 Gbps) 64 15[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 120 1[ ] "el01cn01 EL-C 192.168.10.29 HCA-1" (Could be 5.0 Gbps)
From the output of the iblinkinfo command, note the switch lid value (65, in first column) associated with the Base lid of the compute node port that you noted earlier (121, in the first line):
Determine the gateway switch that corresponds to the switch LID 65 by running the ibswitches command, as in the following example:
Example:
el01cn01# ibswitches
Switch : 0x002128548042c0a0 ports 36 "SUN IB QDR GW switch el01gw03" enhanced port 0 lid 63 lmc 0
Switch : 0x002128547f22c0a0 ports 36 "SUN IB QDR GW switch el01gw02" enhanced port 0 lid 6 lmc 0
Switch : 0x00212856d0a2c0a0 ports 36 "SUN IB QDR GW switch el01gw04" enhanced port 0 lid 65 lmc 0
Switch : 0x00212856d162c0a0 ports 36 "SUN IB QDR GW switch el01gw05" enhanced port 0 lid 64 lmc 0
lid 65 corresponds to gateway switch el01gw04 with GUID 0x00212856d0a2c0a0.
Define a dummy MAC address in the following format:
last3_octets_of_switchGUID : last3_octets_of_computenode_adminIP_in_hex_format
Example:
GUID of switch: 00:21:28:56:d0:a2:c0:a0
Last three octets: a2:c0:a0
Administrative IP of the compute node that requires the VNIC: 192.168.1.1
Last three octets: 168.1.1 (in hexadecimal notation: a8:01:01)
MAC address: a2:c0:a0:a8:01:01
Note:
The dummy MAC address should be unique to the Exalogic network. Only even numbers are supported for the most significant byte of the MAC address (unicast). The above address is an example only.
As ilom-admin, log in to the gateway switch (el01gw04) that you identified in Step 4.
Run the following command to create a VLAN:
hostname# createvlan connector -vlan vlan_ID -pkey default
Example:
e101gw04# createvlan 1A-ETH-3 -vlan 0 -pkey default
Run the following command to create a VNIC:
hostname# createvnic connector -guid compute_node_port_GUID -mac unique_mac_address -pkey default
Example:
el01gw04# createvnic 1A-ETH-3 -guid 0021280001a0a366 -mac a2:c0:a0:a8:01:01 -pkey default
Note:
This new resource is not tagged with any VLAN. At this time, Exalogic uses a single partition (the default partition).
The VNIC is created.
To verify the VNIC, on the switch CLI, run the showvnics command. The following example output is displayed:
ID STATE FLG IOA_GUID NODE IID MAC VLN PKEY GW --- ----- --- ----------------------- --------------------------- ---- ----------------- --- ---- -------- 8 UP N 00:21:28:00:01:A0:A3:66 e101cn01 EL-C 192.168.10.29 0000 a2:c0:a0:a8:01:01 NO ffff 1A-ETH-3
On the compute node, run the following command to display the list of VNICs available on the compute node:
el01cn01# mlx4_vnic_info -l
This command displays the name of the new interface, as seen on the compute node, such as eth4. Note this ID.
Create another VNIC for the same compute node, but using a connector on a different gateway switch. Note the ethX ID of this VNIC too.
It is recommended that you configure the two EoIB interfaces as a bonded interface, such as bond1.
Create interface files for the VNICs on the compute node.
To ensure correct failover behavior, the name of the VNIC interface file and the value of the DEVICE directive in the interface file must not be based on the kernel-assigned ethX interface name (eth4, eth5, and so on). Instead, Oracle recommends that the interface file name and value of the DEVICE directive in the interface file be derived from the EPORT_ID and IOA_PORT values, as follows:
Note:
Any other unique naming scheme is also acceptable.
Run the following command to find the EPORT_ID:
#mlx4_vnic_info -i ethX | grep EPORT_ID
Example:
e101cn01#mlx4_vnic_info -i eth4 | grep EPORT_ID
EPORT_ID 331
Note the EPORT_ID that is displayed, 331 in this example.
Run the following command to find the IOA_PORT:
#mlx4_vnic_info -i ethX | grep IOA_PORT
Example:
e101cn01#mlx4_vnic_info -i eth4 | grep IOA_PORT
IOA_PORT mlx4_0:1
Note the number after the colon (:) in the IOA_PORT value that is displayed, in this case 1.
Build the interface file name and device name by using the following convention:
Interface file name: ifcfg-ethA_B
Device name: ethA_B
A is the EPORT_ID, and B is the number after the colon (:) in the IOA_PORT value.
Example:
Interface file name: ifcfg-eth331_1
Device name: eth331_1
In this example, 331 is the EPORT_ID, and 1 is the value derived from the IOA_PORT.
Create the interface file for the first VNIC, eth4 in the example, by using a text editor such as vi.
Save the file in the /etc/sysconfig/network-scripts directory.
Example for Oracle Linux 6.x or previous version:
# more /etc/sysconfig/network-scripts/ifcfg-eth331_1 DEVICE=eth331_1 BOOTPROTO=none ONBOOT=yes HWADDR=a2:c0:a0:a8:01:01 MASTER=bond1 SLAVE=yes
Example for Oracle Linux 7.x:
# more /etc/sysconfig/network-scripts/ifcfg-eth331_1 DEVICE=eth331_1 BOOTPROTO=none ONBOOT=yes HWADDR=a2:c0:a0:a8:01:01 MASTER=bond1 SLAVE=yes DEVTIMEOUT=30
Make sure that the name of the interface file (ifcfg-eth331_1 in the example) is the name derived in step 12.
For the DEVICE directive, specify the device name (eth331_1 in the example) derived in step 12.
For the HWADDR directive, specify the dummy MAC address created in step 5.
For Oracle Linux 7.x deployments, append DEVTIMEOUT=30 to the interface file.
Note:
The DEVTIMEOUT parameter needs to be added only in the interface configuration files, not in the corresponding bond configuration file.Create an interface file for the second VNIC, say eth5. Be sure to name the interface file and specify the DEVICE directive by using a derived interface name and not the kernel-assigned name, as described earlier. In addition, be sure to specify the relevant dummy MAC address for the HWADDR directive.
After creating the interface files, create the ifcfg-bond1 file. If the file already exists, verify its contents.
Example:
# more /etc/sysconfig/network-scripts/ifcfg-bond1 DEVICE=bond1 IPADDR=192.168.48.128 NETMASK=255.255.255.0 BOOTPROTO=none USERCTL=no TYPE=Ethernet ONBOOT=yes IPV6INIT=no BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000" GATEWAY=192.168.48.1
Bring up the new bond1 interface using the ifup command.
You must also reboot the compute node for the changes to take effect.