To set up Ethernet over InfiniBand connectivity for an Exalogic compute node running Oracle Linux, complete the following steps:
Use an SSH client, such as PuTTY, to log in to a Sun Network QDR InfiniBand Gateway Switch. Oracle recommends that you log in as the root
user. For example, log in to el01gw04
as root
.
At the command prompt, run the following command:
el01gw04# listlinkup | grep Bridge
The following is an example of the output of the listlinkup
command:
Connector 0A-ETH Present Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled) Bridge-0 Port 0A-ETH-2 (Bridge-0-2) up (Enabled) Bridge-0 Port 0A-ETH-3 (Bridge-0-1) up (Enabled) Bridge-0 Port 0A-ETH-4 (Bridge-0-1) up (Enabled) Bridge-0 Port 1A-ETH-1 (Bridge-1-2) down (Enabled) Bridge-0 Port 1A-ETH-2 (Bridge-1-2) down (Enabled) Bridge-0 Port 1A-ETH-3 (Bridge-1-1) up (Enabled) Bridge-0 Port 1A-ETH-4 (Bridge-1-1) up (Enabled)
From this output, identify the uplinks. You can determine that you can use any of the following Ethernet connectors for creating a VNIC:
0A-ETH-1
0A-ETH-2
0A-ETH-3
0A-ETH-4
1A-ETH-3
1A-ETH-4
Note:
This procedure uses 1A-ETH-3
as an example.
Determine GUIDs of the Exalogic compute node that requires the VNIC, as follows:
On the compute node that requires the VNIC, log in as root
, and run the ibstat
command on the command line. For example, log in to el01cn01
as root
.
Example:
el01cn01# ibstat CA 'mlx4_0' CA type: MT26428 Number of ports: 2 Firmware version: 2.7.8100 Hardware version: b0 Node GUID: 0x0021280001a0a364 System image GUID: 0x0021280001a0a367 Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 120 LMC: 0 SM lid: 6 Capability mask: 0x02510868 Port GUID: 0x0021280001a0a365 Link layer: IB Port 2: State: Active Physical state: LinkUp Rate: 40 Base lid: 121 LMC: 0 SM lid: 6 Capability mask: 0x02510868 Port GUID: 0x0021280001a0a366 Link layer: IB
In the output, information about two ports is displayed. Identify the GUID
and Base lid
of the port that you want to use for creating the VNIC.
For the example illustrated in this procedure, we will use the port with GUID 0x0021280001a0a366
and Base lid 121
.
On the same compute node, run the following command to view information about all the active links in the InfiniBand fabric:
hostname# iblinkinfo.pl -R | grep hostname
hostname
is the name of the compute node. You can also specify the bonded IPoIB address of the compute node.
Example:
el01cn01# iblinkinfo.pl -R | grep el01cn01 65 15[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 121 2[ ] "el01cn01 EL-C 192.168.10.29 HCA-1" (Could be 5.0 Gbps) 64 15[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 120 1[ ] "el01cn01 EL-C 192.168.10.29 HCA-1" (Could be 5.0 Gbps)
From the output of the iblinkinfo
command, note the switch lid
value (65
, in first column) associated with the Base lid
of the compute node port that you noted earlier (121
, in the first line):
Determine the gateway switch that corresponds to the switch LID 65
by running the ibswitches
command, as in the following example:
Example:
el01cn01# ibswitches
Switch : 0x002128548042c0a0 ports 36 "SUN IB QDR GW switch el01gw03" enhanced port 0 lid 63 lmc 0
Switch : 0x002128547f22c0a0 ports 36 "SUN IB QDR GW switch el01gw02" enhanced port 0 lid 6 lmc 0
Switch : 0x00212856d0a2c0a0 ports 36 "SUN IB QDR GW switch el01gw04" enhanced port 0 lid 65 lmc 0
Switch : 0x00212856d162c0a0 ports 36 "SUN IB QDR GW switch el01gw05" enhanced port 0 lid 64 lmc 0
lid
65
corresponds to gateway switch el01gw04
with GUID 0x00212856d0a2c0a0
.
Define a dummy MAC address in the following format:
last3_octets_of_switchGUID : last3_octets_of_computenode_adminIP_in_hex_format
Example:
GUID of switch: 00:21:28:56:d0:a2:c0:a0
Last three octets: a2:c0:a0
Administrative IP of the compute node that requires the VNIC: 192.168.1.1
Last three octets: 168.1.1
(in hexadecimal notation: a8:01:01
)
MAC address: a2:c0:a0:a8:01:01
Note:
The dummy MAC address should be unique to the Exalogic network. Only even numbers are supported for the most significant byte of the MAC address (unicast). The above address is an example only.
As ilom-admin
, log in to the gateway switch (el01gw04
) that you identified in Step 4.
Run the following command to create a VLAN:
hostname# createvlan connector -vlan vlan_ID -pkey default
Example:
e101gw04# createvlan 1A-ETH-3 -vlan 0 -pkey default
Run the following command to create a VNIC:
hostname# createvnic connector -guid compute_node_port_GUID -mac unique_mac_address -pkey default
Example:
el01gw04# createvnic 1A-ETH-3 -guid 0021280001a0a366 -mac a2:c0:a0:a8:01:01 -pkey default
Note:
This new resource is not tagged with any VLAN. At this time, Exalogic uses a single partition (the default partition).
The VNIC is created.
To verify the VNIC, on the switch CLI, run the showvnics
command. The following example output is displayed:
ID STATE FLG IOA_GUID NODE IID MAC VLN PKEY GW --- ----- --- ----------------------- --------------------------- ---- ----------------- --- ---- -------- 8 UP N 00:21:28:00:01:A0:A3:66 e101cn01 EL-C 192.168.10.29 0000 a2:c0:a0:a8:01:01 NO ffff 1A-ETH-3
On the compute node, run the following command to display the list of VNICs available on the compute node:
el01cn01# mlx4_vnic_info -l
This command displays the name of the new interface, as seen on the compute node, such as eth4
. Note this ID.
Create another VNIC for the same compute node, but using a connector on a different gateway switch. Note the ethX
ID of this VNIC too.
It is recommended that you configure the two EoIB interfaces as a bonded interface, such as bond1
.
Create interface files for the VNICs on the compute node.
To ensure correct failover behavior, the name of the VNIC interface file and the value of the DEVICE
directive in the interface file must not be based on the kernel-assigned ethX
interface name (eth4
, eth5
, and so on). Instead, Oracle recommends that the interface file name and value of the DEVICE
directive in the interface file be derived from the EPORT_ID
and IOA_PORT
values, as follows:
Note:
Any other unique naming scheme is also acceptable.
Run the following command to find the EPORT_ID
:
#mlx4_vnic_info -i ethX | grep EPORT_ID
Example:
e101cn01#mlx4_vnic_info -i eth4 | grep EPORT_ID
EPORT_ID 331
Note the EPORT_ID
that is displayed, 331
in this example.
Run the following command to find the IOA_PORT
:
#mlx4_vnic_info -i ethX | grep IOA_PORT
Example:
e101cn01#mlx4_vnic_info -i eth4 | grep IOA_PORT
IOA_PORT mlx4_0:1
Note the number after the colon (:) in the IOA_PORT
value that is displayed, in this case 1
.
Build the interface file name and device name by using the following convention:
Interface file name: ifcfg-eth
A
_
B
Device name: eth
A
_
B
A
is the EPORT_ID
, and B
is the number after the colon (:) in the IOA_PORT
value.
Example:
Interface file name: ifcfg-eth
331
_
1
Device name: eth
331
_
1
In this example, 331
is the EPORT_ID
, and 1
is the value derived from the IOA_PORT
.
Create the interface file for the first VNIC, eth4
in the example, by using a text editor such as vi
.
Save the file in the /etc/sysconfig/network-scripts
directory.
Example for Oracle Linux 6.x or previous version:
# more /etc/sysconfig/network-scripts/ifcfg-eth331_1 DEVICE=eth331_1 BOOTPROTO=none ONBOOT=yes HWADDR=a2:c0:a0:a8:01:01 MASTER=bond1 SLAVE=yes
Example for Oracle Linux 7.x:
# more /etc/sysconfig/network-scripts/ifcfg-eth331_1 DEVICE=eth331_1 BOOTPROTO=none ONBOOT=yes HWADDR=a2:c0:a0:a8:01:01 MASTER=bond1 SLAVE=yes DEVTIMEOUT=30
Make sure that the name of the interface file (ifcfg-eth331_1
in the example) is the name derived in step 12.
For the DEVICE
directive, specify the device name (eth331_1
in the example) derived in step 12.
For the HWADDR
directive, specify the dummy MAC address created in step 5.
For Oracle Linux 7.x deployments, append DEVTIMEOUT=30 to the interface file.
Note:
The DEVTIMEOUT parameter needs to be added only in the interface configuration files, not in the corresponding bond configuration file.Create an interface file for the second VNIC, say eth5
. Be sure to name the interface file and specify the DEVICE directive by using a derived interface name and not the kernel-assigned name, as described earlier. In addition, be sure to specify the relevant dummy MAC address for the HWADDR
directive.
After creating the interface files, create the ifcfg-bond1
file. If the file already exists, verify its contents.
Example:
# more /etc/sysconfig/network-scripts/ifcfg-bond1 DEVICE=bond1 IPADDR=192.168.48.128 NETMASK=255.255.255.0 BOOTPROTO=none USERCTL=no TYPE=Ethernet ONBOOT=yes IPV6INIT=no BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000" GATEWAY=192.168.48.1
Bring up the new bond1
interface using the ifup
command.
You must also reboot the compute node for the changes to take effect.