Cabling Several InfiniBand Network Fabric Racks Together

Use this procedure to add another rack to an existing multi-rack system with InfiniBand Network Fabric.

This procedure is for systems with InfiniBand Network Fabric (X8, or earlier).

Set the priority of the current active Subnet Manager Master to 10 on a spine switch.
1. Log in to any InfiniBand switch on the active system.
2. Use the getmaster command to determine that the Subnet Manager Master is running on the spine switch.
  The following example shows that the Subnet Manager Master is running on the spine switch dm01sw-ib1.
```
# getmaster
20100701 11:46:38 OpenSM Master on Switch : 0x0021283a8516a0a0 ports 36 Sun DCS 36
QDR switch dm01sw-ib1.example.com enhanced port 0 lid 1 lmc 0
```
  If the Subnet Manager Master is not running on a spine switch, then perform the following steps:
  1. Use the getmaster command to identify the current location of the Subnet Manager Master.
  2. Log in as the root user on the leaf switch that is the Subnet Manager Master.
  3. Disable Subnet Manager on the switch. The Subnet Manager Master relocates to another switch.
  4. Use the getmaster command to identify the current location of the Subnet Manager Master. If a spine switch is not the Subnet Manager Master, then repeat steps 1.b.ii and 1.b.iii until a spine switch is the Subnet Manager Master.
  5. Enable Subnet Manager on the leaf switches that were disabled during this procedure.
3. Log in to the Subnet Manager Master spine switch.
4. Use the disablesm command to stop the Subnet Manager.
5. Use the setsmpriority 10 command to set the priority to 10.
6. Use the enablesm command to restart the Subnet Manager.
7. Repeat step 1.b to ensure that the Subnet Manager Master is running on the spine switch.
Ensure the new rack is near the existing rack.
The InfiniBand cables must be able to reach the servers in each rack.
Completely shut down the new rack (Rn+1).
Cable the leaf switch in the new rack according to the appropriate table in Multi-Rack Cabling Tables for ZDLRA Rack Models with InfiniBand Network Fabric (X2 to X8).

For example, if rack Rn+1 is R4, then use Table 25-9.

Caution:

Cabling within a live network must be done carefully in order to avoid potentially serious disruptions.

The cabling table that you use for your new InfiniBand topology tells you how to connect ports on the leaf switches to ports on spine switches in order to connect the racks. Some of these ports on the spine switches might be already in use to support the existing InfiniBand topology. In these cases, connect only the cable on the leaf switch in the new rack and stop there for now. Make note of which cables you were not able to terminate.

Do not unplug any cables on the spine switch in the existing rack at this point. Step 5 describes how to re-cable the leaf switches on the existing racks (one leaf switch after the other - while the leaf switch being re-cabled will be powered off), which will free up these currently in-use ports. At that point, you can connect the other end of the cable from the leaf switch in the new rack to the spine switch in the existing rack as indicated in the table.
Complete the following procedure for each of the original racks:
In these steps, Rx represents a rack number from R1 to Rn.
1. Power off leaf switch Rx IB2.
  This causes all servers in the rack to fail over their InfiniBand traffic to Rx IB3.
2. Cable leaf switch Rx IB2 according to Multi-Rack Cabling Tables for ZDLRA Rack Models with InfiniBand Network Fabric (X2 to X8).
3. Power on leaf switch Rx IB2.
4. Wait at least three minutes for Rx IB2 to become completely operational.
  
  To check the switch, log in to the switch and run the ibswitches command. The output should show n*3 switches for IB1, IB2, and IB3 in racks R1, R2, ... Rn.
5. Power off leaf switch Rx IB3.
  This causes all servers in the rack to fail over their InfiniBand traffic to Rx IB2.
6. Cable leaf switch Rx IB3 according to Multi-Rack Cabling Tables for ZDLRA Rack Models with InfiniBand Network Fabric (X2 to X8).
7. Power on leaf switch Rx IB3.
8. Wait at least three minutes for Rx IB3 to become completely operational.
  
  To check the switch, log in to the switch and run the ibswitches command. The output should show n*3 switches for IB1, IB2, and IB3 in racks R1, R2, ... Rn.
All racks should now be rewired according to Multi-Rack Cabling Tables for ZDLRA Rack Models with InfiniBand Network Fabric (X2 to X8).
Power on all the InfiniBand switches in the new rack.
Wait three minutes for the switches to become completely operational.

To check the switch, log in to the switch and run the ibswitches command. The output should show (n+1)*3 switches for IB1, IB2, and IB3 in racks R1, R2, ... Rn+1.
Ensure that the Subnet Manager Master is running on R1 IB1 by running the getmaster command from any switch.
Power on all servers in the new rack (Rn+1).
Log in to spine switch R1 IB1, and lower its priority to 8.
1. Use the disablesm command to stop Subnet Manager.
2. Use the setsmpriority 8 command to set the priority to 8.
3. Use the enablesm command to restart Subnet Manager.
Ensure that the Subnet Manager Master is running on one of the spine switches using the getmaster command from any switch.
Ensure that the Subnet Manager is running on every spine switch by entering the following command from any switch:
```
ibdiagnet -r
```
Each spine switch should show as running in the Summary Fabric SM-state-priority section of the output. If a spine switch is not running, then log in to the switch and enable the Subnet Manager using the enablesm command.
If there are now four or more racks, then log in to the leaf switches in each rack and disable Subnet Manager using the disablesm command.