16.5 Create an IPoIB Partition and Adding Ports

In this example procedure, you are creating a unique, non-default partition named myIPoIB for network isolation on Exalogic's private InfiniBand fabric by configuring a non-default partition key (pkey) value 0x005.

To do so, complete the following steps:

Log in to the InfiniBand switch where master SM is running. For more information, see Before You Begin.
To start the configuration process, run the following command:
```
# smpartition start
```
Create the myIPoIB partition with the pkey 0x005 with full membership by running the following command:
```
# smpartition create -n myIPoIB -pkey 0x8005 -m full -flag ipoib
```
Run the following command to add the compute node port GUIDs, which you noted down in Gather Port GUIDs of Compute Nodes and BridgeX Ports of Gateway Switches, to the myIPoIB partition.
```
# smpartition add -n myIPoIB -port portGUID1 portGUID2
```
In this example, portGUID1 and portGUID2 are the ports that you want to add to the partition. This command example shows a few port entries only. You can add as many ports as necessary. An example port value is 0021280001cef8e3.
If you intend to use the partition for creating vNICs, run the following command to add the gateway switch's BridgeX ports to the myIPoIB partition; otherwise, proceed to the next step. The gateway switch's BridgeX ports were noted down in Gather Port GUIDs of Compute Nodes and BridgeX Ports of Gateway Switches.
```
# smpartition add -n myIPoIB -port BridgeXPort1 BridgeXPort2 BridgeXPort3 BridgeXPort4
```
In this example, BridgeXPort1, BridgeXPort2, BridgeXPort3, and BridgeXPort4 are the BridgeX ports that you want to add to the partition.

Follow these steps to add the ibp0 and the ibp1 network device port GUIDs to the partition:

SSH to the storage appliance and run the following commands to determine the port GUID of the ibp0 network device:

:> configuration
:configuration> net
:configuration net> devices
:configuration net devices> select ibp0
:configuration net devices ibp0> show
Properties:
                         speed = 32000 Mbit/s
                            up = true
                        active = false
                         media = Infiniband
                   factory_mac = not available
                          port = 1
                          guid = 0x212800013e8fbf
configuration net devices ibp0> done

Repeat the previous step with select ibp1 to determine the port GUID of the ibp1 device.
Run the following command to add storage appliance GUIDs to the myIPoIB partition:
```
# smpartition add -n myIPoIB -port ibp0GUID ibp1GUID
```

Run the following command to view the changed partition configuration:
```
# smpartiiton list modified
```
This command displays the new partition with its pkey, ports added to the partition, and membership type.
Run the following command to confirm the partition configuration:
```
# smpartition commit
```
Create interfaces for the ibp0 and the ibp1 network devices and bond them by running these steps:
1. Log in to the Browser User Interface (BUI) of the storage appliance in your Exalogic machine.
2. Under the Configuration tab, select Network.
3. Create a new datalink with the following properties by dragging ibp0 under Devices to the Datalinks column to:
  
  i. In the Name field, enter ibp0.8005, where 8005 is the partition key.
  
  ii. In the Partition Key field, enter the partition key specified for the partition.
  
  iii. Set the Link Mode as Connected Mode.
4. Create a datalink for ibp1 called ibp1.8005 by performing Step Gather Port GUIDs of Compute Nodes and BridgeX Ports of Gateway Switches.
5. Create an interface for the ibp0 network device with the following properties by dragging the ibp0.8005 datalink to the interfaces column:
  
  i. In the Name field, enter ib0.8005, where 8005 is the partition key.
  
  ii. In the Configure with field, select Static Address List.
  
  iii. Enter the IPv4 Address/Mask as 0.0.0.0/8.
6. Create an interface for ibp1 called ib1.8005 by performing Step Gather Port GUIDs of Compute Nodes and BridgeX Ports of Gateway Switches.
7. Bond the interfaces by adding another interface. Click the plus button next to the Interfaces column and enter the following information:
  
  i. In the Name field, enter a name that denotes the IB partition, such as IB_IF_8005.
  
  ii. In the Configure with field, select Static Address List.
  
  iii. Enter the IPv4 Address/Mask to configure as the Infiniband device. You can use any unused IP address. In this example, we will use 192.168.33.15/24.
  
  iv. Check the IP MultiPathing Group box.
The network must be configured on the compute node and the share mounted by following these steps:
1. SSH to a compute node that is a member of the partition as the root user.
2. Determine the active infiniband device by running the following commands on the compute node:
```
# ifconfig ib0
# ifconfig ib1
```
  The active device will have non-zero Rx/Tx bytes.
3. Configure the network for the active device by running the following commands:
```
# echo 0x8005 > /sys/class/net/ib0/create_child
# ifconfig -a | grep -A6 8005
ib0.8005  Link encap:InfiniBand  HWaddr 80:50:05:4C:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
          BROADCAST MULTICAST  MTU:2044  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:256
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
 
# ifconfig ib0.8005 192.168.33.15
```
4. Unmount any mounted shares using the umount command.
5. Mount the share from the remote device. Use the IP specified in the configuration step. In our example we used 192.168.33.15.
```
# mount 192.168.33.15:/export/share_name /mnt/share_name
```
6. Repeat the previous steps for all compute nodes that are members of the partition.