Creating a Basic IPoIB Configuration on a ZFS Storage Appliance

1. Create Datalinks

   Log on to management user interface on the controller head you intend to use as the master. Go to Configuration and then to Network. If you have cabled correctly, two active devices are listed that map onto the cabled IB ports.

   Note

   In active/active configuration both controller heads have 4 active devices: typically ibp0, ibp1, ibp2 and ibp3.

   Drag each of these across to the Datalink menu to create a new datalink for each device. Edit each of these datalinks to provide a datalink name and partition key.

   Figure 8.9 Datalink Configuration

   
   
   

2. Create Interfaces

   Drag each configured datalink across into the Interface menu to create an interface for each datalink that you have defined. Edit each interface to provide a value for the Name field that makes it easy to identify the interface. Add the netmask 0.0.0.0/8 to each interface to prevent the system from probing and testing the routing details behind the network connection. Leave the IP MultiPathing Group unchecked. Do not configure an IP address at this stage.

   Figure 8.10 Interface Configuration

   
   
   

3. Configure Multipathing

   Click the Plus icon in the Interface menu to create an additional interface. This additional interface enables IPMP across the two interfaces you created in the previous step. Enter a value in the Name field that makes it easy to identify the IPMP interface. Assign an IP address from the list of reserved IPs in the Oracle PCA storage network.

   Near the bottom of the window, select the IP MultiPathing Group check box. Select the two previously configured interfaces, mark the first one as Active and the second one as Standby. The selected IP address will be configured on the active interface until a failover to the standby interface occurs.

   Note

   In active/active configuration you create 2 IPMP interfaces: one across interfaces ibp0 and ibp2, and the other across ibp1 and ibp3. This ensures redundancy across the two PCIe InfiniBand expansion cards. Assign an IP address to each IPMP interface. The configuration is replicated to the other storage controller.

   Figure 8.11 Multipathing (IPMP) Configuration

   
   
   

4. Apply Connectivity Configuration

   Click the Apply button to commit the connectivity configuration you performed up to this point. When you select any device, datalink or interface associated with your configuration, all related items are highlighted in the user interface, as shown in the screenshot below. All other items unrelated to the configuration in this procedure have been blurred out in the image.

   Figure 8.12 Final Connectivity Configuration

   
   
   

5. Verify Management Cluster Configuration

   The external storage setup proposed in this chapter has an active/passive cluster configuration. Consequently all storage resources for Oracle PCA have to be available at one particular IP address at any time. The IP address and the storage resources are owned by the active head in the management cluster, and the standby head is ready to take over the IP address and expose the same storage resources if the active head should fail.

   On the active storage controller, go to Configuration and then to Cluster. The active/passive cluster configuration is illustrated in the screenshot below. All resources involved in the Oracle PCA external storage setup are owned by the active head: the storage appliance management interface, the IPMP interface configured specifically for Oracle PCA external storage connectivity, and the storage pool where the LUNs will be created.

   In this setup the entire configuration is applied on the active controller and automatically replicated on the standby controller. Since all resources are owned by the active head, there is no need to log on to the standby head and make any configuration changes.

   Note

   In active/active configuration both storage controllers take ownership of the storage resources in their storage pool. Each storage controller presents its storage resources to Oracle PCA using separate redundant network paths. The active path at any given moment is determined through multipathing.

   Figure 8.13 Management Cluster Configuration

   
   
   

6. Configure iSCSI Targets

   On the active controller, go to Configuration, SAN and then to ISCSI. Click the Plus icon in the Targets menu to create a new iSCSI target. Enter a value in the Alias field that makes it easy to identify the iSCSI target. Select the IPMP interface you configured specifically for Oracle PCA external storage connectivity. Click OK to create the new target.

   Figure 8.14 Create iSCSI Target

   
   
   

   Drag the iSCSI target that you have created into the Target Group area to create a target group. Edit the target group, give it an appropriate name, and make sure the IQN of the correct iSCSI target is selected. Click OK to save your changes.

   Figure 8.15 Create iSCSI Target Group

   
   
   
   Note

   In active/active configuration you must create an iSCSI target and target group on each storage controller. In the target configuration select the two IPMP interfaces you created.

   The iSCSI initiators and initiator group, which are configured in the steps below, are automatically replicated to the second storage controller.

7. Configure iSCSI Initiators

   First, you need to retrieve the IQN of each compute node you wish to grant access to the external storage. Log on to the master management node of your Oracle PCA and proceed as follows:

   1. Using the CLI, list all compute nodes.

      [root@ovcamn05r1 ~]# pca-admin list compute-node
      
      Compute_Node  IP_Address    Provisioning_Status  ILOM_MAC            Provisioning_State
      ------------  ----------    -------------------  --------            ------------------
      ovcacn09r1    192.168.4.7   RUNNING              00:10:e0:3f:82:75   running
      ovcacn13r1    192.168.4.11  RUNNING              00:10:e0:3f:87:73   running
      ovcacn26r1    192.168.4.13  RUNNING              00:10:e0:3e:46:db   running
      ovcacn12r1    192.168.4.10  RUNNING              00:10:e0:3f:8a:c7   running
      ovcacn08r1    192.168.4.6   RUNNING              00:10:e0:3f:84:df   running
      ovcacn27r1    192.168.4.14  RUNNING              00:10:e0:3f:9f:13   running
      ovcacn07r1    192.168.4.5   RUNNING              00:10:e0:3f:75:73   running
      ovcacn11r1    192.168.4.9   RUNNING              00:10:e0:3f:83:23   running
      ovcacn10r1    192.168.4.8   RUNNING              00:10:e0:3f:89:83   running
      ovcacn14r1    192.168.4.12  RUNNING              00:10:e0:3f:8b:5d   running
      -----------------
      10 rows displayed
      
      Status: Success

   2. SSH into each compute node and display the contents of the file initiatorname.iscsi.

      [root@ovcamn05r1 ~]# ssh ovcacn07r1
      root@ovcacn07r1's password:
      [root@ovcacn07r1 ~]# cat /etc/iscsi/initiatorname.iscsi
      InitiatorName=iqn.1988-12.com.oracle:a72be49151
      
      [root@ovcacn07r1 ~]# exit
      logout
      Connection to ovcacn07r1 closed.
      
      [root@ovcamn05r1 ~]#

      Tip

      Using SSH to connect to each of the listed compute nodes is the fastest way to obtain the IQNs. However, they can also be copied from the Oracle VM Manager user interface, in the Storage Initiator perspective for each server.

   3. Copy the IQN of the compute node and use it to define a corresponding iSCSI initiator in the ZFS Storage Appliance user interface.

   Click on the Initiators link to define the iSCSI initiators for the compute nodes that you wish to expose LUNs to. Click the Plus icon to identify a new iSCSI initiator. Enter the compute node Initiator IQN and an Alias that makes it easy to identify the iSCSI initiator. Repeat this for each compute node so that all initiators appear in the list.

   Figure 8.16 Identify iSCSI Initiators

   
   
   

   Drag the one of the iSCSI initiators that you have created into the Initiator Group area to create a new initiator group. Edit the initiator group, give it an appropriate name, and make sure that all IQNs of the compute nodes that you want to make a member of this initiator group are selected. Click OK to save your changes.

   Figure 8.17 Create iSCSI Initiator Group

   
   
   

8. Apply iSCSI Target and Initiator Configuration

   Click the Apply button to commit the iSCSI target and initiator configuration you performed up to this point. When you select any initiator or initiator group associated with your configuration, all related items are highlighted in the user interface, as shown in the screenshot below. Other items unrelated to the configuration in this procedure have been blurred out in the image.

   Figure 8.18 Final iSCSI Configuration

   
   
   

9. Create a LUN

   For easy management it is good practice to organize storage resources in separate projects. Create a project first for your Oracle PCA external storage, and then add LUNs as you need them.

   On the active storage controller, go to Shares. On the left hand side, click the Plus icon to create a new Project. In the Create Project dialog box, enter a Name to make it easy to identify the project. Click Apply to save the new project.

   Figure 8.19 Create ZFS Storage Project

   
   
   

   In the navigation pane on the left hand side, select the project you just created. In the Shares window, first select LUNs,and then click the Plus icon to create a new LUN as part of this project.

   Fill out the LUN properties, select the iSCSI target group and initiator group you created earlier, and enter a Name that makes it easy to identify the LUN. Click Apply to add the LUN to the selected project.

   Figure 8.20 Create LUN

   
   
   
   Note

   In active/active configuration each storage controller owns a storage pool. You must create a storage project for use with Oracle PCA on each storage controller. Associate the LUNs you create with the iSCSI initiator group containing the Oracle PCA compute nodes, and with the iSCSI target group of the selected storage controller. As a result, the LUNs in question are owned by that storage controller. For optimal performance the ownership of storage resources should be balanced between both active storage controllers.