Skip Headers
Oracle® Exalogic Elastic Cloud Machine Owner's Guide
Release EL X2-2, X3-2, and X4-2

E18478-18
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
PDF · Mobi · ePub

10 Configuring Ethernet Over InfiniBand

This chapter describes how to set up Ethernet over InfiniBand (EoIB) network configuration for your Exalogic machine.

For more information about cabling a gateway switch to an external data center 10 Gb Ethernet switch, see Section 6.6.6, "Connectivity Between Exalogic Machine and External LAN Through Sun Network QDR InfiniBand Gateway Switch". For information about cable and transceiver requirements, see Section 6.6.6.3, "Transceiver and Cable Requirements".

This chapter contains the following topics:

10.1 Introduction to Virtual NICs (VNICs)

A virtual NIC or VNIC maps an Ethernet connector on the Sun Network QDR InfiniBand Gateway Switch to a network interface within the compute node. A connector (0A-ETH-1 to 0A-ETH-4, and1A-ETH-1 to 1A-ETH-4) hosts a 10 GbE port. In the Exalogic machine, on each Sun Network QDR InfiniBand gateway switch, you can have a maximum of eight 10 GbE uplinks. In most scenarios, the number of 10 GbE uplinks is less than 8.

One or more VNICs can be assigned to a compute node. If more than one VNIC is assigned, they can be different connectors on the same Sun Network QDR InfiniBand Gateway Switch or on different Sun Network QDR InfiniBand Gateway Switches. The default configuration is to map one connector on each of the two switches and to bond or balance the pair for increased availability.

Note:

In Exalogic, you create VNICs in Sun Network QDR InfiniBand Gateway Switch manual mode.

VNIC Resource Limit

On each Sun Network QDR InfiniBand Gateway Switch there can be a maximum of 1K VNICs per logical gateway. Logical gateway denotes either a single external Ethernet port or a defined group of external Ethernet ports. The 1K limit is shared by all VLANs on the logical gateway and each defined VLAN ID also consumes one of the 1K VNICs. For more information on VLANs see Chapter 11, "Virtual LANs (VLANs)".

To find out how many VNICs are currently defined, use the showvnics command and for VLANs, use showvlan.

10.2 Setting Up Ethernet Over InfiniBand (EoIB) on Oracle Linux

To set up Ethernet over InfiniBand connectivity for an Exalogic compute node running Oracle Linux, complete the following steps:

  1. Use an SSH client, such as PuTTY, to log in to a Sun Network QDR InfiniBand Gateway Switch. Oracle recommends that you log in as the root user. For example, log in to el01gw04 as root.

  2. At the command prompt, run the following command:

    el01gw04# listlinkup | grep Bridge
    

    The following is an example of the output of the listlinkup command:

    Connector 0A-ETH Present
      Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled)
      Bridge-0 Port 0A-ETH-2 (Bridge-0-2) up (Enabled)
      Bridge-0 Port 0A-ETH-3 (Bridge-0-1) up (Enabled)
      Bridge-0 Port 0A-ETH-4 (Bridge-0-1) up (Enabled)
      Bridge-0 Port 1A-ETH-1 (Bridge-1-2) down (Enabled)
      Bridge-0 Port 1A-ETH-2 (Bridge-1-2) down (Enabled)
      Bridge-0 Port 1A-ETH-3 (Bridge-1-1) up (Enabled)
      Bridge-0 Port 1A-ETH-4 (Bridge-1-1) up (Enabled)
    

    From this output, identify the uplinks. You can determine that you can use any of the following Ethernet connectors for creating a VNIC:

    • 0A-ETH-1

    • 0A-ETH-2

    • 0A-ETH-3

    • 0A-ETH-4

    • 1A-ETH-3

    • 1A-ETH-4

      Note:

      This procedure uses 1A-ETH-3 as an example.

  3. Determine GUIDs of the Exalogic compute node that requires the VNIC, as follows:

    1. On the compute node that requires the VNIC, log in as root, and run the ibstat command on the command line. For example, log in to el01cn01 as root.

      Example:

      el01cn01# ibstat
      CA 'mlx4_0'
              CA type: MT26428
              Number of ports: 2
              Firmware version: 2.7.8100
              Hardware version: b0
              Node GUID: 0x0021280001a0a364
              System image GUID: 0x0021280001a0a367
              Port 1:
                      State: Active
                      Physical state: LinkUp
                      Rate: 40
                      Base lid: 120
                      LMC: 0
                      SM lid: 6
                      Capability mask: 0x02510868
                      Port GUID: 0x0021280001a0a365
                      Link layer: IB
              Port 2:
                      State: Active
                      Physical state: LinkUp
                      Rate: 40
                      Base lid: 121
                      LMC: 0
                      SM lid: 6
                      Capability mask: 0x02510868
                      Port GUID: 0x0021280001a0a366
                      Link layer: IB
      

      In the output, information about two ports is displayed. Identify the GUID and Base lid of the port that you want to use for creating the VNIC.

      For the example illustrated in this procedure, we will use the port with GUID 0x0021280001a0a366 and Base lid 121.

    2. On the same compute node, run the following command to view information about all the active links in the InfiniBand fabric:

      hostname# iblinkinfo.pl -R | grep hostname
      

      hostname is the name of the compute node. You can also specify the bonded IPoIB address of the compute node.

      Example:

      el01cn01# iblinkinfo.pl -R | grep el01cn01
      65   15[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>    121   2[  ] "el01cn01 EL-C 192.168.10.29 HCA-1" (Could be 5.0 Gbps)
      64   15[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>    120   1[  ] "el01cn01 EL-C 192.168.10.29 HCA-1" (Could be 5.0 Gbps)
      

      From the output of the iblinkinfo command, note the switch lid value (65, in first column) associated with the Base lid of the compute node port that you noted earlier (121, in the first line):

  4. Determine the gateway switch that corresponds to the switch LID 65 by running the ibswitches command, as in the following example:

    Example:

    el01cn01# ibswitches
    Switch  : 0x002128548042c0a0 ports 36 "SUN IB QDR GW switch el01gw03" enhanced port 0 lid 63 lmc 0
    Switch  : 0x002128547f22c0a0 ports 36 "SUN IB QDR GW switch el01gw02" enhanced port 0 lid 6 lmc 0
    Switch  : 0x00212856d0a2c0a0 ports 36 "SUN IB QDR GW switch el01gw04" enhanced port 0 lid 65 lmc 0
    Switch  : 0x00212856d162c0a0 ports 36 "SUN IB QDR GW switch el01gw05" enhanced port 0 lid 64 lmc 0
    

    lid 65 corresponds to gateway switch el01gw04 with GUID 0x00212856d0a2c0a0.

  5. Define a dummy MAC address in the following format:

    last3_octets_of_switchGUID : last3_octets_of_computenode_adminIP_in_hex_format
    

    Example:

    GUID of switch: 00:21:28:56:d0:a2:c0:a0

    Last three octets: a2:c0:a0

    Administrative IP of the compute node that requires the VNIC: 192.168.1.1

    Last three octets: 168.1.1 (in hexadecimal notation: a8:01:01)

    MAC address: a2:c0:a0:a8:01:01

    Note:

    The dummy MAC address should be unique to the Exalogic network. Only even numbers are supported for the most significant byte of the MAC address (unicast). The above address is an example only.

  6. As ilom-admin, log in to the gateway switch (el01gw04) that you identified in Step 4.

  7. Run the following command to create a VLAN:

    hostname# createvlan connector -vlan vlan_ID -pkey default
    

    Example:

    e101gw04# createvlan 1A-ETH-3 -vlan 0 -pkey default
    
  8. Run the following command to create a VNIC:

    hostname# createvnic connector -guid compute_node_port_GUID -mac unique_mac_address -pkey default
    

    Example:

    el01gw04# createvnic 1A-ETH-3 -guid 0021280001a0a366 -mac a2:c0:a0:a8:01:01 -pkey default
    

    Note:

    This new resource is not tagged with any VLAN. At this time, Exalogic uses a single partition (the default partition).

    The VNIC is created.

  9. To verify the VNIC, on the switch CLI, run the showvnics command. The following example output is displayed:

    Surrounding text describes vnicinfo1.png.
  10. On the compute node, run the following command to display the list of VNICs available on the compute node:

    el01cn01# mlx4_vnic_info -l
    

    This command displays the name of the new interface, as seen on the compute node, such as eth4. Note this ID.

  11. Create another VNIC for the same compute node, but using a connector on a different gateway switch. Note the ethX ID of this VNIC too.

    It is recommended that you configure the two EoIB interfaces as a bonded interface, such as bond1.

  12. Create interface files for the VNICs on the compute node.

    To ensure correct failover behavior, the name of the VNIC interface file and the value of the DEVICE directive in the interface file must not be based on the kernel-assigned ethX interface name (eth4, eth5, and so on). Instead, Oracle recommends that the interface file name and value of the DEVICE directive in the interface file be derived from the EPORT_ID and IOA_PORT values, as follows:

    Note:

    Any other unique naming scheme is also acceptable.

    1. Run the following command to find the EPORT_ID:

      #mlx4_vnic_info -i ethX | grep EPORT_ID
      

      Example:

      e101cn01#mlx4_vnic_info -i eth4 | grep EPORT_ID
      EPORT_ID     331
      

      Note the EPORT_ID that is displayed, 331 in this example.

    2. Run the following command to find the IOA_PORT:

      #mlx4_vnic_info -i ethX | grep IOA_PORT
      

      Example:

      e101cn01#mlx4_vnic_info -i eth4 | grep IOA_PORT
      IOA_PORT     mlx4_0:1
      

      Note the number after the colon (:) in the IOA_PORT value that is displayed, in this case 1.

    3. Build the interface file name and device name by using the following convention:

      Interface file name: ifcfg-ethA_B

      Device name: ethA_B

      A is the EPORT_ID, and B is the number after the colon (:) in the IOA_PORT value.

      Example:

      Interface file name: ifcfg-eth331_1

      Device name: eth331_1

      In this example, 331 is the EPORT_ID, and 1 is the value derived from the IOA_PORT.

  13. Create the interface file for the first VNIC, eth4 in the example, by using a text editor such as vi.

    Save the file in the /etc/sysconfig/network-scripts directory.

    Example:

    # more /etc/sysconfig/network-scripts/ifcfg-eth331_1
    DEVICE=eth331_1
    BOOTPROTO=none 
    ONBOOT=yes 
    HWADDR=a2:c0:a0:a8:01:01
    MASTER=bond1 
    SLAVE=yes 
    
    • Make sure that the name of the interface file (ifcfg-eth331_1 in the example) is the name derived in step 12.

    • For the DEVICE directive, specify the device name (eth331_1 in the example) derived in step 12.

    • For the HWADDR directive, specify the dummy MAC address created in step 5.

  14. Create an interface file for the second VNIC, say eth5. Be sure to name the interface file and specify the DEVICE directive by using a derived interface name and not the kernel-assigned name, as described earlier. In addition, be sure to specify the relevant dummy MAC address for the HWADDR directive.

  15. After creating the interface files, create the ifcfg-bond1 file. If the file already exists, verify its contents.

    Example:

    # more /etc/sysconfig/network-scripts/ifcfg-bond1
    DEVICE=bond1 
    IPADDR=192.168.48.128 
    NETMASK=255.255.255.0 
    BOOTPROTO=none 
    USERCTL=no 
    TYPE=Ethernet 
    ONBOOT=yes 
    IPV6INIT=no 
    BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000" 
    GATEWAY=192.168.48.1
    
  16. Bring up the new bond1 interface using the ifup command.

    You must also reboot the compute node for the changes to take effect.

10.3 Setting Up Ethernet Over InfiniBand (EoIB) on Oracle Solaris 11 Express

To set up Ethernet over InfiniBand connectivity for Exalogic compute nodes running Oracle Solaris 11 Express, complete the following steps:

  1. Use an SSH client, such as PuTTY, to log in to a Sun Network QDR InfiniBand Gateway Switch as a root. For example, log in to el01gw04 as root.

  2. At the command prompt, run the following command:

    el01gw04# listlinkup | grep Bridge
    

    A section of the output of this command is as follows:

    Connector 0A-ETH Present
      Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled)
      Bridge-0 Port 0A-ETH-2 (Bridge-0-2) up (Enabled)
      Bridge-0 Port 0A-ETH-3 (Bridge-0-1) up (Enabled)
      Bridge-0 Port 0A-ETH-4 (Bridge-0-1) up (Enabled)
      Bridge-0 Port 1A-ETH-1 (Bridge-1-2) down (Enabled)
      Bridge-0 Port 1A-ETH-2 (Bridge-1-2) down (Enabled)
      Bridge-0 Port 1A-ETH-3 (Bridge-1-1) up (Enabled)
      Bridge-0 Port 1A-ETH-4 (Bridge-1-1) up (Enabled)
    

    From this example, identify the uplinks. You can determine that you can use any of the following Ethernet connectors for creating a VNIC:

    • 0A-ETH-1

    • 0A-ETH-2

    • 0A-ETH-3

    • 0A-ETH-4

    • 1A-ETH-3

    • 1A-ETH-4

      Note:

      This example procedure uses 1A-ETH-3.

  3. Determine GUIDs of an Exalogic compute node as follows:

    1. On the compute node that requires the VNIC, log in as root, and run the dladm show-ib command on the command line. For example, log in to el01cn02 as root. This command displays port information, as in the following example output:

      el01cn02# dladm show-ib
      LINK     HCAGUID            PORTGUID           PORT  STATE  PKEYS
      ibp0     21280001A0A694     21280001A0A695     1     up     FFFF
      ibp1     21280001A0A694     21280001A0A696     2     up     FFFF
      

      In the output, information about two ports is displayed. From this output, you must determine which port GUID to use. This example procedure uses the port GUID 21280001A0A695 (port 1).

    2. On the same compute node, run the following command on the command line to report information about all active links in the InfiniBand fabric:

      el01cn02# iblinkinfo.pl -R | grep hostname
      

      Where hostname is the name of the compute node. For example, el01cn02.

      The following is the example output of this command:

      el01cn02# iblinkinfo.pl -R | grep el01cn02
      65   15[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>    121   2[  ] "el01cn02 EL-C 192.168.10.29 HCA-1" (Could be 5.0 Gbps)
      64   15[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>    120   1[  ] "el01cn02 EL-C 192.168.10.29 HCA-1" (Could be 5.0 Gbps)
      

      From this example output, note down the switch lid values. The switch lid of port 1 is 64 (the first column in the output). The switch lid of port 2 is 65.

  4. Determine which gateway switch is associated with the switch lids by comparing the first column of the iblinkinfo output to the lid value of the ibswitches command as follows:

    1. On the compute node, run the ibswitches command on the command line. The example output of this command is as follows:

      el01cn02# ibswitches
      Switch  : 0x002128548042c0a0 ports 36 "SUN IB QDR GW switch el01gw03" enhanced port 0 lid 63 lmc 0
      Switch  : 0x002128547f22c0a0 ports 36 "SUN IB QDR GW switch el01gw02" enhanced port 0 lid 6 lmc 0
      Switch  : 0x00212856d0a2c0a0 ports 36 "SUN IB QDR GW switch el01gw04" enhanced port 0 lid 65 lmc 0
      Switch  : 0x00212856d162c0a0 ports 36 "SUN IB QDR GW switch el01gw05" enhanced port 0 lid 64 lmc 0
      
    2. In this example output, identify the switches that lid values 64 and 65 are associated with. In this example, the switch lid 64 of the gateway switch el01gw05 with GUID 0x00212856d162c0a0 is associated with port 1 of the HCA in the compute node el01cn02.

      Note:

      This example procedure uses LID 64 of this gateway switch.

  5. Define a dummy MAC address in the following format:

    <last three octets from el01gw05 switch ib GUID> : <last three octets of the administrative IP of the compute node in hexadecimal format>

    Example:

    GUID of switch el01gw05: 00:21:28:56:d1:62:c0:a0

    Last three octets of the switch GUID: 62:c0:a0

    Administrative IP address of compute node: 192.168.1.5

    Last three octets of the compute node's IP address: 168.1.5

    Last three octets in hexadecimal notation: a8:01:05.

    MAC address of the VNIC: 62:c0:a0:a8:01:05

    Note:

    Each MAC address should be unique. Only even numbers are supported for the most significant byte of the MAC address (unicast). The above address is an example only.

  6. As root, log in to el01gw05 that you identified in Step 4. Use its IP address or host name to log in.

  7. Upon login, run the following command to create a VNIC:

    el01gw05# createvnic 1A-ETH-3 -guid 00:21:28:00:01:A0:A6:95 -mac 62:c0:a0:a8:01:05 -pkey default
    

    Note:

    This new resource is not tagged with any VLAN.

    A VNIC is created.

  8. To verify the VNIC, on the switch CLI, run the showvnics command. The following example output is displayed:

    Surrounding text describes vnicinfo2.png.
  9. On the compute node, run the following command to display the list of VNICs available on the compute node:

    el01cn02# dladm show-phys | grep eoib
    

    This command displays the name of the new interface, as seen on the compute node, such as eoib0. It also indicates the state of the interface.

    Note:

    You may repeat the above steps to create more network- administered tagless VNICs on the same compute node as long as a unique {ETH connector, port GUID} tuple is chosen each time. When this second VNIC is configured in the same manner, the VNIC is seen on the compute node (for example, as the eoib1 interface). It is recommended that you configure these two Ethernet over InfiniBand (EoIB) interfaces in an IPMP group, such as bond1.

    To create a host-administered VNIC on a {ETH connector, port GUID} tuple with a network-administered tagless VNIC already created on it, complete the steps described in Section 11.5, "Oracle Solaris 11 Express: Creating VNICs and Associating Them with VLANs".

  10. Before creating the bond1 IPMP group, you should save the VNIC configuration to a file, so the VNIC is persistent. To do so, create a file named hostname.eoib0 in the /etc directory by using a text editor. Enter the IP address of the eoib0 interface and the name in the following format:

    IP_address interface_name

    For example, if eoib0 is associated with the IP 192.168.3.4, then enter the following in the hostname.eoib0 file:

    192.168.3.4 eoib0

  11. To configure eoib1 and eoib2 in an IPMP group for high availability purposes, complete the following steps:

    1. Plumb the IPMP group named bond1:

      el01cn02# ifconfig bond1 ipmp IP_ADDRESS_FOR_BOND1 netmask NETMASK_FOR_BOND1 up
      
    2. Plumb the EoIB interfaces:

      el01cn02# ifconfig eoib0 plumb group bond1 up
      
      el01cn02# ifconfig eoib1 plumb group bond1 standby up
      

      Note:

      In this example, eoib0 is the primary EoIB interface, and eoib1 is the standby interface.

    3. Create a file named hostname.bond1 in the /etc directory by using a text editor. Enter the IP address of the bond1 interface and the name in the following format:

      IP_address interface_name

      For example, if bond1 is associated with the IP 192.168.3.10, then enter the following in the hostname.bond1 file:

      192.168.3.10 bond1

    4. Set up the configuration to persist across reboot:

      el01cn02# echo "ipmp group bond1 IP_ADDRESS_FOR_BOND1 up" > /etc/hostname.bond1
      
      el01cn02# echo "group bond1 standby" > /etc/hostname.eoib1
      

10.4 Setting Up Ethernet Over InfiniBand (EoIB) on Oracle Solaris 11.1

You can set up Ethernet over InfiniBand connectivity for Exalogic compute nodes running Oracle Solaris 11.1 by doing the following:

  1. Use an SSH client, such as PuTTY, to log in to a Sun Network QDR InfiniBand Gateway Switch as a root. For example, log in to el01gw04 as root.

  2. At the command prompt, run the following command:

    el01gw04# listlinkup | grep Bridge
    

    A section of the output of this command is as follows:

    Connector 0A-ETH Present
      Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled)
      Bridge-0 Port 0A-ETH-2 (Bridge-0-2) up (Enabled)
      Bridge-0 Port 0A-ETH-3 (Bridge-0-1) up (Enabled)
      Bridge-0 Port 0A-ETH-4 (Bridge-0-1) up (Enabled)
      Bridge-0 Port 1A-ETH-1 (Bridge-1-2) down (Enabled)
      Bridge-0 Port 1A-ETH-2 (Bridge-1-2) down (Enabled)
      Bridge-0 Port 1A-ETH-3 (Bridge-1-1) up (Enabled)
      Bridge-0 Port 1A-ETH-4 (Bridge-1-1) up (Enabled)
    

    From this example, identify the uplinks. In this example, you can use any of the following Ethernet connectors for creating a VNIC:

    • 0A-ETH-1

    • 0A-ETH-2

    • 0A-ETH-3

    • 0A-ETH-4

    • 1A-ETH-3

    • 1A-ETH-4

      Note:

      This example procedure uses 1A-ETH-3.

  3. Determine GUIDs of an Exalogic compute node as follows:

    1. On the compute node that requires the VNIC, log in as root, and run the dladm show-ib command on the command line. For example, log in to el01cn02 as root. This command displays port information, as in the following example output:

      el01cn02# dladm show-ib
      LINK     HCAGUID            PORTGUID           PORT  STATE  PKEYS
      ibp0     21280001A0A694     21280001A0A695     1     up     FFFF
      ibp1     21280001A0A694     21280001A0A696     2     up     FFFF
      

      In the output, information about two ports is displayed. From this output, you must determine which port GUID to use. This example procedure uses the port GUID 21280001A0A695 (port 1).

    2. On the same compute node, run the following command on the command line to report information about all active links in the InfiniBand fabric:

      el01cn02# iblinkinfo.pl -R | grep hostname
      

      Where hostname is the name of the compute node. For example, el01cn02.

      The following is the example output of this command:

      el01cn02# iblinkinfo.pl -R | grep el01cn02
      65   15[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>    121   2[  ] "el01cn02 EL-C 192.168.10.29 HCA-1" (Could be 5.0 Gbps)
      64   15[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>    120   1[  ] "el01cn02 EL-C 192.168.10.29 HCA-1" (Could be 5.0 Gbps)
      

      From this example output, note down the switch lid values. The switch lid of port 1 is 64 (the first column in the output). The switch lid of port 2 is 65.

  4. Determine which gateway switch is associated with the switch lids by comparing the first column of the iblinkinfo output to the lid value of the ibswitches command as follows:

    1. On the compute node, run the ibswitches command on the command line. The example output of this command is as follows:

      el01cn02# ibswitches
      Switch  : 0x002128548042c0a0 ports 36 "SUN IB QDR GW switch el01gw03" enhanced port 0 lid 63 lmc 0
      Switch  : 0x002128547f22c0a0 ports 36 "SUN IB QDR GW switch el01gw02" enhanced port 0 lid 6 lmc 0
      Switch  : 0x00212856d0a2c0a0 ports 36 "SUN IB QDR GW switch el01gw04" enhanced port 0 lid 65 lmc 0
      Switch  : 0x00212856d162c0a0 ports 36 "SUN IB QDR GW switch el01gw05" enhanced port 0 lid 64 lmc 0
      
    2. In this example output, identify the switches that lid values 64 and 65 are associated with. In this example, the switch lid 64 of the gateway switch el01gw05 with GUID 0x00212856d162c0a0 is associated with port 1 of the HCA in the compute node el01cn02.

      Note:

      This example procedure uses LID 64 of this gateway switch.

  5. Define a dummy MAC address in the following format:

    <last three octets from el01gw05 switch ib GUID> : <last three octets of the administrative IP of the compute node in hexadecimal format>

    Example:

    GUID of switch el01gw05: 00:21:28:56:d1:62:c0:a0

    Last three octets of the switch GUID: 62:c0:a0

    Administrative IP address of compute node: 192.168.1.5

    Last three octets of the compute node's IP address: 168.1.5

    Last three octets in hexadecimal notation: a8:01:05.

    MAC address of the VNIC: 62:c0:a0:a8:01:05

    Note:

    Each MAC address should be unique. Only even numbers are supported for the most significant byte of the MAC address (unicast). The above address is an example only.

  6. As root, log in to el01gw05 that you identified in Step 4. Use its IP address or host name to log in.

  7. Upon login, to permit the configuration of VNICs, run the following command:

    el01gw05# allowhostconfig
    
  8. To create a VLAN, run the following command:

    e101gw05# createvlan 1A-ETH-3 -vlan 1706 -pkey default
    
  9. Note the ID of the VLAN you created by running the showvlan command as follows:

    # showvlan
      Connector/LAG  VLN   PKEY
      -------------  ---   ----
      1A-ETH-3        0    ffff
      1A-ETH-3        1706 ffff
    

    In this example, the VLAN ID is 1706.

  10. Run the following command to create a VNIC on the switch:

    el01gw05# createvnic 1A-ETH-3 -guid 00:21:28:00:01:A0:A6:95 -mac 62:c0:a0:a8:01:05 -pkey default
    

    Note:

    This new resource is not tagged with any VLAN.

    A VNIC is created.

  11. To verify that the VNIC was created, run the showvnics command. The following example output is displayed:

    Surrounding text describes vnicinfo2.png.
  12. On the compute node, run the following command to display the list of VNICs available on the compute node:

    el01cn02# dladm show-phys | grep eoib
    

    This command displays the name of the new interface, as seen on the compute node, such as eoib0. Note the corresponding link, such as net7. It also displays the state of the interface.

    Note:

    You may repeat the above steps to create more network- administered tagless VNICs on the same compute node as long as a unique {ETH connector, port GUID} tuple is chosen each time. When this second VNIC is configured in the same manner, the VNIC is seen on the compute node (for example, as the eoib1 interface with the link net8). It is recommended that you configure these two Ethernet over InfiniBand (EoIB) interfaces in an IPMP group, such as bond1.

    To create a host-administered VNIC on a {ETH connector, port GUID} tuple with a network-administered tagless VNIC already created on it, complete the steps described in Section 11.5, "Oracle Solaris 11 Express: Creating VNICs and Associating Them with VLANs".

  13. Create another VNIC for the same compute node, using a connector on a different gateway switch, by following steps 1 to 12. Note the name of this interface and its corresponding link. For example, eoib1 interface with the link net8.

  14. Delete the following files:

    • /etc/hostname.bond1

    • /etc/hostname.eoib0

    • /etc/hostname.eoib0

  15. Restart the compute node by running the reboot command.

  16. Create the VNIC you created in step 8 again on the compute node by running the following command:

    hostname# dladm create-vnic -l link_name [-v vlan_id] interface_name
    

    Example:

    el01cn02# dladm create-vnic -l net7 eoib0
    el01cn02# dladm create-vnic -l net8 eoib1
    

    If you are creating a VLAN tagged VNIC, use the -v option to add the VLAN ID as follows:

    el01cn02# dladm create-vnic -l net7 -v 1706 eoib0
    el01cn02# dladm create-vnic -l net8 -v 1706 eoib1
    
  17. You can verify if the VNICs were created by using the dladm show-vnic command as follows:

    hostname# dladm show-vnic
    
  18. To configure eoib0 and eoib1 in an IPMP group for high availability purposes, do the following:

    1. Identify the data links associated with the VNICs you created on the InfiniBand switch by running the following command:

      el01cn02# dladm show-phys -m
      

      Identify the link names associated with the VNICs you created, such as net7 and net8.

    2. Create the IPMP group by running the following command:

      hostname# ipadm create-ipmp bond_name
      

      Example:

      el01cn02# ipadm create-ipmp bond1
      
    3. Create the IP interfaces for the two links you noted in step a by running the ipadm create-ip command as follows:

      hostname# ipadm create-ip link_name
      

      Example:

      el01cn02# ipadm create-ip net7
      el01cn02# ipadm create-ip net8
      
    4. Create interfaces for the VNICs you created in step 16 by running the following commands:

      hostname# ipadm create-ip interface_name 
      

      Example:

      el01cn02# ipadm create-ip eoib0
      el01cn02# ipadm create-ip eoib1
      
    5. Set one of the interfaces as a standby for the bonded interface, by running the following command:

      hostname# ipadm set-ifprop -p standby=on -m ip interface_name
      

      Example:

      e101cn02# ipadm set-ifprop -p standby=on -m ip eoib1
      
    6. Add the two interfaces to the ipmp bond you created in step b, by running the following command:

      hostname# ipadm add-ipmp -i interface_name1 -i interface_name2 bond_name
      

      Example:

      e101cn02# ipadm add-ipmp -i eoib0 -i eoib1 bond1
      
    7. Set an IP address for the bonded interface you created, by running the following command:

      hostname# ipadm create-addr –T static –a local=ipv4_address/CIDR_netmask bond_name/v4
      

      Example:

      e101cn02# ipadm create-addr –T static –a local=10.100.44.68/22 bond1/v4
      
    8. Verify that your bonded interface is up, by running the following command:

      hostname# ipadm show-if
      IFNAME     CLASS    STATE    ACTIVE OVER
      lo0        loopback ok       yes    --
      net0       ip       ok       yes    --
      net4       ip       ok       yes    --
      net8       ip       down     no     --
      net9       ip       down     no     --
      bond0_0    ip       ok       yes    --
      bond0_1    ip       ok       no     --
      bond1     ipmp     ok       yes    eoib1 eoib0
      eoib1      ip       ok       no     --
      eoib0      ip       ok       yes    --
      
    9. Verify that your bonded interface was given an IP address by running the following command:

      # ipadm show-addr
      ADDROBJ           TYPE     STATE        ADDR
      lo0/v4            static   ok           127.0.0.1/8
      net0/v4           static   ok           138.3.2.87/21
      net4/v4           static   ok           169.254.182.77/24
      bond0/v4          static   ok           192.168.14.101/24
      bond1/v4          static   ok           138.3.48.35/22
      bond1/v4a         static   ok           138.3.51.1/22
      lo0/v6            static   ok           ::1/128
      net0/v6           addrconf ok          fe80::221:28ff:fed7:e944/10