Configuring the Oracle SPARC SuperCluster InfiniBand Switches to Add the ZFS Storage Appliance

In this procedure, the GUIDs of the ZFS Storage Appliance Infiniband HBA ports are added to the existing Oracle SPARC SuperCluster InfiniBand configuration. By adding these ports and using a partition key of 8503, communication between the two devices can occur.

1. Log on to the Oracle SPARC SuperCluster InfiniBand spine switch as root. By default, the spine switch is given a hostname of <sscid>sw- ib1, where <sscid> is the prefix name given to the entire Oracle SPARC SuperCluster system. In the following example, the <sscid> is aiessc.

login as: root
root@aiesscsw-ib1's password:
Last login: Tue Sep 25 08:19:01 2013 from dhcp-brm-bl5-204-3e
east-10-135-75-254.usdhcp.oraclecorp.com

2. Enter the command enablesm to verify that the switch is running Subnet Manager (or this command will start Subnet Manager).<br/>

[root@aiesscsw-ib1 ~]# enablesm
opensm (pid 15906) is already running...
Starting partitiond daemon
/usr/local/util/partitiond is already running
(You may also perform a 'restart' if wanted)

3. Enter the command getmaster to verify that this is the master switch of the configuration. If the master switch is not running on the spine switch, log out and log in to the designated master switch for the remainder of this procedure.<br/>

[root@aiesscsw-ib1 ~]# getmaster
Local SM enabled and running
20130913 10:16:51 Master SubnetManager on sm lid 13 sm guid
0x2128e8ac27a0a0 : SUN DCS 36P QDR aiesscsw-ib1.us.oracle.com
[root@aiesscsw-ib1 ~]#

4. Back up the switch configuration according the documented backup procedures (http://docs.oracle.com/cd/E26698_01/index.html).

5. Enter the command smpartition list active to verify that partition key 0x0503 is assigned to partition name "sto" (sto = 0x0503).<br/> The partition key was set to 8503 on the ZFS Storage Appliance datalinks, but the InfiniBand switch reports 0503. This is intentional because the InfiniBand protocol reserves the most significant bit (0x8000) of the hexadecimal partition key (pkey) for its own use. Therefore, pkeys 0x8503 and 0x0503 are the same.<br/>

[root@aiesscsw-ib1 ~]# smpartition list active
# Sun DCS IB partition config file
# This file is generated, do not edit
#! version_number : 11
Default=0x7fff, ipoib : ALL_CAS=full, ALL_SWITCHES=full, SELF=
full;
SUN_DCS=0x0001, ipoib : ALL_SWITCHES=full;
ic1s10 = 0x0501,ipoib,defmember=full:
0x0021280001ef30f7,
0x0021280001ef33bf,
0x0021280001ef30b7,
0x0021280001ef314b;
ic2s10 = 0x0502,ipoib,defmember=full:
0x0021280001ef30f8,
0x0021280001ef33c0,
0x0021280001ef30b8,
0x0021280001ef314c;
sto = 0x0503,ipoib,defmember=full:
0x0021280001ef43f8,
0x0021280001ef43b7,
0x0021280001cf90c0,
0x0021280001ef43bb,
...more...

6. Add the ZFS Storage Appliance to the InfiniBand configuration:

• Enter the command smpartition start to start a reconfiguration session.<br/>

# smpartition start<br/>
[root@aiesscsw-ib1 ~]# smpartition start

• Enter the command smpartition add to add the eight new GUIDs to the configuration.<br/>

# smpartition add -n sto -port <GUID1> <GUID2> <GUID3> ...  <GUID8><br/>
[root@aiesscsw-ib1 ~]# smpartition add -n sto -port
21280001ef43bb 21280001ef43bc 21280001cf90bf 21280001cf90c0
21280001ef43f7 21280001ef43f8 21280001ef43b7 21280001ef43b8

• Enter the command smpartition list modified to verify the new GUIDs have been added correctly.<br/>

# smpartition list modified<br/>
[root@aiesscsw-ib1 ~]# smpartition list modified
# Sun DCS IB partition config file
<nowki># This file is generated, do not edit
#! version_number : 11
Default=0x7fff, ipoib : ALL_CAS=full, ALL_SWITCHES=full, SELF=
full;
SUN_DCS=0x0001, ipoib : ALL_SWITCHES=full;
ic1s10 = 0x0501,ipoib,defmember=full:
0x0021280001ef30f7,
0x0021280001ef33bf,
0x0021280001ef30b7,
0x0021280001ef314b;
ic2s10 = 0x0502,ipoib,defmember=full:
0x0021280001ef30f8,
0x0021280001ef33c0,
0x0021280001ef30b8,
0x0021280001ef314c;
sto = 0x0503,ipoib,defmember=full:
0x0021280001ef43f8,
0x0021280001ef43b7,
0x0021280001cf90c0,
0x0021280001ef43bb,
0x0021280001ef43bc,
0x0021280001cf90bf,
0x0021280001ef43b8,
0x0021280001ef43f7,
0x0021280001ef3048,
0x0021280001ef30af,
0x0021280001ef30f8,
0x0021280001ef30f7,
0x0021280001ef33c0,
0x0021280001ef33bf,
0x0021280001ef30cc,
0x0021280001ef342b,
0x0021280001ef30b8,
0x0021280001ef30b7,
0x0021280001ef314c,
0x0021280001ef314b,
0x0021280001efec65,
0x0021280001efec66,
0x0021280001efecb1,
0x0021280001efecb2;

• Enter the command smpartition commit to apply the new configuration and propagate configuration changes to all InfiniBand switches in the configuration.<br/>

# smpartition commit<br/>
[root@aiesscsw-ib1 ~]# smpartition commit
[root@aiesscsw-ib1 ~]#

7. Log off the InfiniBand switch.

8. Back up the InfiniBand configuration according to the documented backup procedures (http://docs.oracle.com/cd/E26698_01/index.html).