This section includes the following procedures to set up EoIB on Oracle Solaris:
You can set up Ethernet over InfiniBand connectivity for Exalogic compute nodes running Oracle Solaris 11.1 by doing the following:
Use an SSH client, such as PuTTY, to log in to a Sun Network QDR InfiniBand Gateway Switch as a root. For example, log in to el01gw04 as root.
At the command prompt, run the following command:
el01gw04# listlinkup | grep Bridge
A section of the output of this command is as follows:
Connector 0A-ETH Present Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled) Bridge-0 Port 0A-ETH-2 (Bridge-0-2) up (Enabled) Bridge-0 Port 0A-ETH-3 (Bridge-0-1) up (Enabled) Bridge-0 Port 0A-ETH-4 (Bridge-0-1) up (Enabled) Bridge-0 Port 1A-ETH-1 (Bridge-1-2) down (Enabled) Bridge-0 Port 1A-ETH-2 (Bridge-1-2) down (Enabled) Bridge-0 Port 1A-ETH-3 (Bridge-1-1) up (Enabled) Bridge-0 Port 1A-ETH-4 (Bridge-1-1) up (Enabled)
From this example, identify the uplinks. In this example, you can use any of the following Ethernet connectors for creating a VNIC:
0A-ETH-1
0A-ETH-2
0A-ETH-3
0A-ETH-4
1A-ETH-3
1A-ETH-4
Note:
This example procedure uses 1A-ETH-3.
Determine GUIDs of an Exalogic compute node as follows:
On the compute node that requires the VNIC, log in as root, and run the dladm show-ib command on the command line. For example, log in to el01cn02 as root. This command displays port information, as in the following example output:
el01cn02# dladm show-ib LINK HCAGUID PORTGUID PORT STATE PKEYS ibp0 21280001A0A694 21280001A0A695 1 up FFFF ibp1 21280001A0A694 21280001A0A696 2 up FFFF
In the output, information about two ports is displayed. From this output, you must determine which port GUID to use. This example procedure uses the port GUID 21280001A0A695 (port 1).
On the same compute node, run the following command on the command line to report information about all active links in the InfiniBand fabric:
el01cn02# iblinkinfo.pl -R | grep hostname
Where hostname is the name of the compute node. For example, el01cn02.
The following is the example output of this command:
el01cn02# iblinkinfo.pl -R | grep el01cn02 65 15[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 121 2[ ] "el01cn02 EL-C 192.168.10.29 HCA-1" (Could be 5.0 Gbps) 64 15[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 120 1[ ] "el01cn02 EL-C 192.168.10.29 HCA-1" (Could be 5.0 Gbps)
From this example output, note down the switch lid values. The switch lid of port 1 is 64 (the first column in the output). The switch lid of port 2 is 65.
Determine which gateway switch is associated with the switch lids by comparing the first column of the iblinkinfo output to the lid value of the ibswitches command as follows:
On the compute node, run the ibswitches command on the command line. The example output of this command is as follows:
el01cn02# ibswitches Switch : 0x002128548042c0a0 ports 36 "SUN IB QDR GW switch el01gw03" enhanced port 0 lid 63 lmc 0 Switch : 0x002128547f22c0a0 ports 36 "SUN IB QDR GW switch el01gw02" enhanced port 0 lid 6 lmc 0 Switch : 0x00212856d0a2c0a0 ports 36 "SUN IB QDR GW switch el01gw04" enhanced port 0 lid 65 lmc 0 Switch : 0x00212856d162c0a0 ports 36 "SUN IB QDR GW switch el01gw05" enhanced port 0 lid 64 lmc 0
In this example output, identify the switches that lid values 64 and 65 are associated with. In this example, the switch lid 64 of the gateway switch el01gw05 with GUID 0x00212856d162c0a0 is associated with port 1 of the HCA in the compute node el01cn02.
Note:
This example procedure uses LID 64 of this gateway switch.
Define a dummy MAC address in the following format:
<last three octets from el01gw05 switch ib GUID> : <last three octets of the administrative IP of the compute node in hexadecimal format>
Example:
GUID of switch el01gw05: 00:21:28:56:d1:62:c0:a0
Last three octets of the switch GUID: 62:c0:a0
Administrative IP address of compute node: 192.168.1.5
Last three octets of the compute node's IP address: 168.1.5
Last three octets in hexadecimal notation: a8:01:05.
MAC address of the VNIC: 62:c0:a0:a8:01:05
Note:
Each MAC address should be unique. Only even numbers are supported for the most significant byte of the MAC address (unicast). The above address is an example only.
As root, log in to el01gw05 that you identified in Step 4. Use its IP address or host name to log in.
Upon login, to permit the configuration of VNICs, run the following command:
el01gw05# allowhostconfig
To create a VLAN, run the following command:
e101gw05# createvlan 1A-ETH-3 -vlan 1706 -pkey default
Note the ID of the VLAN you created by running the showvlan command as follows:
# showvlan Connector/LAG VLN PKEY ------------- --- ---- 1A-ETH-3 0 ffff 1A-ETH-3 1706 ffff
In this example, the VLAN ID is 1706.
Run the following command to create a VNIC on the switch:
el01gw05# createvnic 1A-ETH-3 -guid 00:21:28:00:01:A0:A6:95 -mac 62:c0:a0:a8:01:05 -pkey default
Note:
This new resource is not tagged with any VLAN.
A VNIC is created.
To verify that the VNIC was created, run the showvnics command. The following example output is displayed:
ID STATE FLG IOA_GUID NODE IID MAC VLN PKEY GW --- ----- --- ----------------------- --------------------------- ---- ----------------- --- ---- -------- 0 UP N 00:21:28:00:01:A0:A6:95 e101cn01 EL-C 192.168.10.29 0000 62:c0:a0:a8:01:05 NO ffff 1A-ETH-3
On the compute node, run the following command to display the list of VNICs available on the compute node:
el01cn02# dladm show-phys | grep eoib
This command displays the name of the new interface, as seen on the compute node, such as eoib0. Note the corresponding link, such as net7. It also displays the state of the interface.
Note:
You may repeat the above steps to create more network- administered tagless VNICs on the same compute node as long as a unique {ETH connector, port GUID} tuple is chosen each time. When this second VNIC is configured in the same manner, the VNIC is seen on the compute node (for example, as the eoib1 interface with the link net8). It is recommended that you configure these two Ethernet over InfiniBand (EoIB) interfaces in an IPMP group, such as bond1.
To create a host-administered VNIC on a {ETH connector, port GUID} tuple with a network-administered tagless VNIC already created on it, complete the steps described in Oracle Solaris: Creating VNICs and Associating Them with VLANs.
Create another VNIC for the same compute node, using a connector on a different gateway switch, by following steps 1 to 12. Note the name of this interface and its corresponding link. For example, eoib1 interface with the link net8.
Delete the following files:
/etc/hostname.bond1
/etc/hostname.eoib0
/etc/hostname.eoib0
Restart the compute node by running the reboot command.
Create the VNIC you created in step 8 again on the compute node by running the following command:
hostname# dladm create-vnic -l link_name [-v vlan_id] interface_name
Example:
el01cn02# dladm create-vnic -l net7 eoib0 el01cn02# dladm create-vnic -l net8 eoib1
If you are creating a VLAN tagged VNIC, use the -v option to add the VLAN ID as follows:
el01cn02# dladm create-vnic -l net7 -v 1706 eoib0 el01cn02# dladm create-vnic -l net8 -v 1706 eoib1
You can verify if the VNICs were created by using the dladm show-vnic command as follows:
hostname# dladm show-vnic
To configure eoib0 and eoib1 in an IPMP group for high availability purposes, do the following:
Identify the data links associated with the VNICs you created on the InfiniBand switch by running the following command:
el01cn02# dladm show-phys -m
Identify the link names associated with the VNICs you created, such as net7 and net8.
Create the IPMP group by running the following command:
hostname# ipadm create-ipmp bond_name
Example:
el01cn02# ipadm create-ipmp bond1
Create the IP interfaces for the two links you noted in step 18.a by running the ipadm create-ip command as follows:
hostname# ipadm create-ip link_name
Example:
el01cn02# ipadm create-ip net7 el01cn02# ipadm create-ip net8
Create interfaces for the VNICs you created in step 16 by running the following commands:
hostname# ipadm create-ip interface_name
Example:
el01cn02# ipadm create-ip eoib0 el01cn02# ipadm create-ip eoib1
Set one of the interfaces as a standby for the bonded interface, by running the following command:
hostname# ipadm set-ifprop -p standby=on -m ip interface_name
Example:
e101cn02# ipadm set-ifprop -p standby=on -m ip eoib1
Add the two interfaces to the ipmp bond you created in step 18.b, by running the following command:
hostname# ipadm add-ipmp -i interface_name1 -i interface_name2 bond_name
Example:
e101cn02# ipadm add-ipmp -i eoib0 -i eoib1 bond1
Set an IP address for the bonded interface you created, by running the following command:
hostname# ipadm create-addr –T static –a local=ipv4_address/CIDR_netmask bond_name/v4
Example:
e101cn02# ipadm create-addr –T static –a local=10.100.44.68/22 bond1/v4
Verify that your bonded interface is up, by running the following command:
hostname# ipadm show-if
IFNAME CLASS STATE ACTIVE OVER
lo0 loopback ok yes --
net0 ip ok yes --
net4 ip ok yes --
net8 ip down no --
net9 ip down no --
bond0_0 ip ok yes --
bond0_1 ip ok no --
bond1 ipmp ok yes eoib1 eoib0
eoib1 ip ok no --
eoib0 ip ok yes --
Verify that your bonded interface was given an IP address by running the following command:
# ipadm show-addr
ADDROBJ TYPE STATE ADDR
lo0/v4 static ok 127.0.0.1/8
net0/v4 static ok 138.3.2.87/21
net4/v4 static ok 169.254.182.77/24
bond0/v4 static ok 192.168.14.101/24
bond1/v4 static ok 138.3.48.35/22
bond1/v4a static ok 138.3.51.1/22
lo0/v6 static ok ::1/128
net0/v6 addrconf ok fe80::221:28ff:fed7:e944/10