|Oracle® Big Data Appliance Owner's Guide
Release 1 (1.0.3)
Part Number E25960-05
|PDF · Mobi · ePub|
This chapter discusses the following sections:
A virtual network interface card (VNIC) maps an Ethernet connector on the Sun Network QDR InfiniBand Gateway Switch to a network interface within the server. A connector (
1A-ETH-4) hosts a 10 GbE port. On each Sun Network QDR InfiniBand Gateway Switch on Oracle Big Data Appliance, you can have a maximum of eight 10 GbE uplinks. In most scenarios, the number of 10 GbE uplinks is less than eight.
The network interface on the server, which is produced by creating a VNIC on the Sun Network QDR InfiniBand Gateway Switch, uses the
x naming convention, such as
eth9. You can assign multiple VNICs to a server. In that case, they can be different connectors on the same Sun Network QDR InfiniBand Gateway Switch or on different ones. The default configuration maps one connector on each of the two switches, and bonds or balances the pair for increased availability.
In Oracle Big Data Appliance, you manually create VNICs in Sun Network QDR InfiniBand Gateway Switch.
To set up Ethernet over InfiniBand connectivity:
At the command prompt, run the following command:
This is an example of the output of this command:
Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled) Bridge-0 Port 0A-ETH-2 (Bridge-0-2) down (Enabled) Bridge-0 Port 0A-ETH-3 (Bridge-0-1) down (Enabled) Bridge-0 Port 0A-ETH-4 (Bridge-0-1) down (Enabled) Bridge-1 Port 1A-ETH-1 (Bridge-1-2) down (Enabled) Bridge-1 Port 1A-ETH-2 (Bridge-1-2) down (Enabled) Bridge-1 Port 1A-ETH-3 (Bridge-1-1) down (Enabled) Bridge-1 Port 1A-ETH-4 (Bridge-1-1) down (Enabled)
In this example, the uplink is
0A-ETH-1; all others are down.
Determine the globally unique identifiers (GUIDs) of an Oracle Big Data Appliance server as follows:
On the server that requires the VNIC, log in as
root, and run the
ibstat command on the command line. For example, log in to bda1node01 as
root. This command displays port information, as shown in the following example output:
# ibstat CA 'mlx4_0' CA type: MT26428 Number of ports: 2 Firmware version: 2.9.1000 Hardware version: b0 Node GUID: 0x0021280001cf4b56 System image GUID: 0x0021280001cf4b59 Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 137 LMC: 0 SM lid: 147 Capability mask: 0x02510868 Port GUID: 0x0021280001cf4b57 Link layer: IB Port 2: State: Active Physical state: LinkUp Rate: 40 Base lid: 138 LMC: 0 SM lid: 147 Capability mask: 0x02510868 Port GUID: 0x0021280001cf4b58 Link layer: IB
In the output, information for two ports is displayed. From this example, you must determine which port GUID to use. Before doing so, write down the two Base lid values shown in the output. In the example output, Port 1 has a Base lid value of
137, and Port 2 has a Base lid value of
On the same server, run the following command on the command line to report information about all active links in the InfiniBand fabric:
In this command, hostname is the name of the server, such as
bda1node01. You can also specify the bonded IP over InfiniBand (IPoIB) address of the server.
The following is the example output of this command:
148 6[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 138 2[ ] "bda1node01 HCA-1" ( ) 149 6[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 137 1[ ] "bda1node01 HCA-1" ( )
From this example output, note the switch lid values. Port 2 base lid of
138 is associated with switch lid
148 (the first column in the output). Port 1 base lid of
137 is associated with switch lid
On the server, issue the
ibswitches command. The example output of this command follows.
# ibswitches Switch : 0x002128df0f0ac0a0 ports 36 "SUN IB QDR GW switch bda1sw-ib3 10.133.43.37" enhanced port 0 lid 148 lmc 0 Switch : 0x0021284690eea0a0 ports 36 "SUN DCS 36P QDR bda1sw-ib1 10.133.43.35" enhanced port 0 lid 147 lmc 0 Switch : 0x002128df348ac0a0 ports 36 "SUN IB QDR GW switch bda1sw-ib2 10.133.43.36" enhanced port 0 lid 149 lmc 0
In this example output, identify the switches that lid values
149 are associated with.
148 is associated with gateway switch
bda1sw-ib3 and GUID
149 is associated with gateway switch
bda1sw-ib2 and GUID
0x002128df348ac0a0. The following examples use this switch.
last three octets from bda1sw-ib2 switch ib GUID
last three octets of the administrative IP of the server in hexadecimal
For example, the
bda1sw-ib2 switch GUID is
0x002128df348ac0a0). The last three octets are
Assume that the administrative IP of the server that requires the VNIC is
192.168.1.1. The last three octets of the IP address are
168.1.1. Convert each decimal to hexadecimal, and it is
Then you can define the MAC address as follows:
Note:Ensure that the MAC address is unique. Only even numbers are supported for the most significant byte of the MAC address (unicast). The address used here is an example only.
root, log in to the switch identified in Step 4. Use its IP address or host name to log in. In this example, the switch is
This command does not associate the VNIC with a VLAN. See Chapter 11 for information about creating VLANs.
Oracle Big Data Appliance uses a single partition (the default partition).
To verify the VNIC, run the
showvnics command. The following example output is displayed:
# mlx4_vnic_info --list eth8
You must configure these two Ethernet over InfiniBand (EoIB) interfaces as a bonded interface named
bondeth0. This procedure requires that you manually create three files:
ifcfg-eth8: Describes the
ifcfg-eth9: Describes the
ifcfg-bondeth0: Bonds the two connections in active-passive failover mode
To create a bonded interface from the EoIB interfaces:
Log in as
root to the same server where you created the VNICs and change to the
Using a text editor such as
vi, create or edit a file named
ifcfg-eth8 with the following contents. Replace hostname with the name of the server, and address with the dummy MAC address created in Step 5 of the previous procedure.
#ifcfg_eth8 for host hostname DEVICE=eth8 MASTER=bondeth0 SLAVE=yes ONBOOT=yes MTU=65520 BOOTPROTO=none HOTPLUG=no IPV6INIT=no HWADDR=address
Create or edit a second file named
eth9, using its dummy MAC address.
Create or edit a file named
ifcfg-bondeth0 with the following parameters. Verify that the settings are correct for your network.
#ifcfg-bondeth0 for server bda1node01 generated by BDAeiob.g DEVICE=bondeth0 TYPE=ETHERNET BROADCAST=220.127.116.11 ONBOOT=yes #MTU=65520 BOOTPROTO=none NETMASK=255.255.255.0 IPV6INIT=no NETWORK=18.104.22.168 IPADDR=203.0.113.10 BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000"
Start the new
bondeth0 interface, and restart the server for the changes to take effect.