You can create a partition for EoIB (both inbound and outbound) and associate the partition's pkey with a VLAN and VNIC on the edge network.
Note:
The port GUID values, MAC addresses, VLAN IDs, compute node names, gateway switch names, and Ethernet connector names used in this procedure are examples only.
At the command prompt on one of the gateway switches, run the following command:
el01gw04# listlinkup | grep Bridge
The following is an example of the output of the lislinkup command:
Connector 0A-ETH Present Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled) Bridge-0 Port 0A-ETH-2 (Bridge-0-2) up (Enabled) Bridge-0 Port 0A-ETH-3 (Bridge-0-1) up (Enabled) Bridge-0 Port 0A-ETH-4 (Bridge-0-1) up (Enabled) Bridge-0 Port 1A-ETH-1 (Bridge-1-2) down (Enabled) Bridge-0 Port 1A-ETH-2 (Bridge-1-2) down (Enabled) Bridge-0 Port 1A-ETH-3 (Bridge-1-1) up (Enabled) Bridge-0 Port 1A-ETH-4 (Bridge-1-1) up (Enabled)
From this example, identify the uplinks. You can determine that you can use any of the following Ethernet connectors for creating a VNIC:
0A-ETH-1
0A-ETH-2
0A-ETH-3
0A-ETH-4
1A-ETH-3
1A-ETH-4
Note:
This procedure uses 1A-ETH-3 as an example.
Determine GUIDs of the Exalogic compute node the requires the VNIC as follows:
On the compute node that requires the VNIC, log in as root, and run the ibstat command on the command line. For example, log in to el01cn01 as root.
Example:
el01cn01# ibstat
CA 'mlx4_0'
CA type: MT26428
Number of ports: 2
Firmware version: 2.7.8100
Hardware version: b0
Node GUID: 0x0021280001a0a364
System image GUID: 0x0021280001a0a367
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 120
LMC: 0
SM lid: 6
Capability mask: 0x02510868
Port GUID: 0x0021280001a0a365
Link layer: IB
Port 2:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 121
LMC: 0
SM lid: 6
Capability mask: 0x02510868
Port GUID: 0x0021280001a0a366
Link layer: IB
In the output, information about two ports is displayed. Identify the GUID and Base lid of the port that you want to use for creating the VNIC.
For the example illustrated in this procedure, we will use the port with GUID 0x0021280001a0a366 and Base lid 121.
On the same compute node, run the following command to view information about all the active links in the InfiniBand fabric:
hostname# iblinkinfo.pl -R | grep hostname
hostname is the name of the compute node. You can also specify the bonded IPoIB address of the compute node.
Example:
el01cn01# iblinkinfo.pl -R | grep el01cn01 65 15[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 121 2[ ] "el01cn01 EL-C 192.168.10.29 HCA-1" (Could be 5.0 Gbps) 64 15[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 120 1[ ] "el01cn01 EL-C 192.168.10.29 HCA-1" (Could be 5.0 Gbps)
From the output of the iblinkinfo command, note the switch lid value (65, in first column) associated with the Base lid of the compute node port that you noted earlier (121, in the first line):
Determine the gateway switch that corresponds to the switch lid 65 by running the ibswitches command, as in the following example:
Example:
el01cn01# ibswitches
Switch : 0x002128548042c0a0 ports 36 "SUN IB QDR GW switch el01gw03" enhanced port 0 lid 63 lmc 0
Switch : 0x002128547f22c0a0 ports 36 "SUN IB QDR GW switch el01gw02" enhanced port 0 lid 6 lmc 0
Switch : 0x00212856d0a2c0a0 ports 36 "SUN IB QDR GW switch el01gw04" enhanced port 0 lid 65 lmc 0
Switch : 0x00212856d162c0a0 ports 36 "SUN IB QDR GW switch el01gw05" enhanced port 0 lid 64 lmc 0
lid 65 corresponds to gateway switch el01gw04 with GUID 0x00212856d0a2c0a0.
Define a dummy MAC address in the following format:
last3_octets_of_switchGUID : last3_octets_of_computenode_adminIP_in_hex_format
Example:
GUID of switch: 00:21:28:56:d0:a2:c0:a0
Last three octets: a2:c0:a0
Administrative IP of the compute node that requires the VNIC: 192.168.1.1
Last three octets: 168.1.1 (in hexadecimal notation: a8:01:01)
MAC address: a2:c0:a0:a8:01:01
Note:
The dummy MAC address should be unique to the Exalogic network. Only even numbers are supported for the most significant byte of the MAC address (unicast). The above address is an example only.
Ensure that you have noted down all port GUIDs and BridgeX ports.
Log in to the InfiniBand switch where master SM is running. For more information, see Before You Begin.
Run the following command to start the configuration process:
# smpartition start
Run the following command to create a myEoIB partition with the pkey 0x005 with a full membership:
# smpartition create -n myEoIB -pkey 0x005 -m full
Run the following command to add port GUIDs and BridgeX ports, which you noted down in Gather Port GUIDs of Compute Nodes and BridgeX Ports of Gateway Switches, to the myEoIB partition:
# smpartition add -n myEoIB -port port_guid1 port_guid2 bridgex_port1 bridgex_port2
Where port_guid1, port_guid2, bridgex_port1, and bridgex_port2 are the ports that you want to add to the partition. This command example shows a few port entries only. You can add as many ports as necessary. An example port value is 0021280001cef8e3.
Run the following command to view the changed partition configuration:
# smpartition list modified
This command displays the new partition with its pkey, ports added to the partition, and membership type.
Run the following command to confirm the partition configuration:
# smpartition commit
The myEoIB partition with 0x005 pkey is created.
Log in to the gateway switch interface as root, and run the following commands:
# createvlan 1A-ETH-3 -vlan 10 -pkey 0x005
Where 1A-ETH-3 is the Ethernet connector on the gateway switch, 10 is the VLAN identifier, and 0x005 is the partition key that you created earlier.
To verify, run the following command:
# showvlan
The following information is displayed:
Connector/LAG VLN PKEY -------------- --- ----- 1A-ETH-3 10 0x005 0A-ETH-1 11 ffff
As root, log in to el01gw04 that you identified in Step 4. Use its IP address or host name to log in.
Upon login, run the following command to create a VNIC:
# createvnic 1A-ETH-3 -GUID 00212856d0a2c0a0 -mac a2:c0:a0:a8:01:01 -vlan 10 -pkey 0x005
Where 1A-ETH-3 is the Ethernet connector, 00:21:28:56:d0:a2:c0:a0 is the GUID, a2:c0:a0:a8:01:01 is the dummy MAC address defined in Step 4, 10 is the VLAN identifier, and 0x005 is the partition key that you created earlier.
This example creates a VNIC, such as eth4 (on Oracle Linux) or eoib0 (on Oracle Solaris) associated with VLAN 10 associated with a partition with 0x005 as the pkey.
Run the following command to verify the VNICs:
# showvnics
The following message is displayed:
ID STATE FLG IOA_GUID NODE IID MAC VLN PKEY GW --- ----- --- ----------------------- --------------------------- ---- ----------------- --- ----- -------- 8 UP N 00:21:28:00:01:A0:A3:66 e101cn01 EL-C 192.168.10.29 0000 a2:c0:a0:a8:01:01 10 0x005 1A-ETH-3
Tip:
After creating the interfaces, you can run the ifconfig command with the -a option to verify the MAC address on the compute node. For example, to verify the new interface and its MAC address, run the following command on the Oracle Linux compute node for which the VNIC was created:
# ifconfig -a eth4
The output of this command shows the HWADDR, which is the MAC address you defined for the VNIC in Step 5.
On the compute node, run the following command to display the list of VNICs available on the compute node:
el01cn01# mlx4_vnic_info -l
This command displays the name of the new interface, as seen on the compute node, such as eth4. Note this ID.
Create another VNIC for the same compute node, but using a connector on a different gateway switch. Note the ethX ID of this VNIC too.
It is recommended that you configure the two EoIB interfaces as a bonded interface, such as bond1.
Create interface files for the VNICs on the compute node.
To ensure correct failover behavior, the name of the VNIC interface file and the value of the DEVICE directive in the interface file must not be based on the kernel-assigned ethX interface name (eth4, eth5, and so on). Instead, Oracle recommends that the interface file name and value of the DEVICE directive in the interface file be derived from the EPORT_ID and IOA_PORT values, as follows:
Note:
Any other unique naming scheme is also acceptable.
Run the following command to find the EPORT_ID:
#mlx4_vnic_info -i ethX | grep EPORT_ID
Example:
e101cn01#mlx4_vnic_info -i eth4 | grep EPORT_ID
EPORT_ID 331
Note the EPORT_ID that is displayed, 331 in this example.
Run the following command to find the IOA_PORT:
#mlx4_vnic_info -i ethX | grep IOA_PORT
Example:
e101cn01#mlx4_vnic_info -i eth4 | grep IOA_PORT
IOA_PORT mlx4_0:1
Note the number after the colon (:) in the IOA_PORT value that is displayed, in this case 1.
Build the interface file name and device name by using the following convention:
Interface file name: ifcfg-ethA_B
Device name: ethA_B
A is the EPORT_ID, and B is the number after the colon (:) in the IOA_PORT value.
Example:
Interface file name: ifcfg-eth331_1
Device name: eth331_1
In this example, 331 is the EPORT_ID, and 1 is the value derived from the IOA_PORT.
Create the interface file for the first VNIC, eth4 in the example, by using a text editor such as vi.
Save the file in the /etc/sysconfig/network-scripts directory.
Example:
# more /etc/sysconfig/network-scripts/ifcfg-eth331_1 DEVICE=eth331_1 BOOTPROTO=none ONBOOT=yes HWADDR=a2:c0:a0:a8:01:01 MASTER=bond1 SLAVE=yes
Create an interface file for the second VNIC, say eth5. Be sure to name the interface file and specify the DEVICE directive by using a derived interface name and not the kernel-assigned name, as described earlier. In addition, be sure to specify the relevant dummy MAC address for the HWADDR directive.
After creating the interface files, create the ifcfg-bond1 file. If the file already exists, verify its contents.
Example:
# more /etc/sysconfig/network-scripts/ifcfg-bond1 DEVICE=bond1 IPADDR=192.168.48.128 NETMASK=255.255.255.0 BOOTPROTO=none USERCTL=no TYPE=Ethernet ONBOOT=yes IPV6INIT=no BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000" GATEWAY=192.168.48.1
Restart the network services by running the following command:
# service network restart
Bring up the new bond1 interface using the ifup command.
You must also reboot the compute node for the changes to take effect.