Cabling Two InfiniBand Network Fabric Racks Together

Use this procedure to cable together two racks with InfiniBand Network Fabric.

This procedure assumes that the racks are adjacent to each other.

In the procedure, the existing rack is R1, and the new rack is R2.

1. Set the priority of the current, active Subnet Manager Master to 10 on the spine switch, as follows:

   1. Log in to any RDMA Network Fabric switch on the active system.

   2. Use the getmaster command to determine that the Subnet Manager Master is running on the spine switch.

   3. Log in to the spine switch.

   4. Use the disablesm command to stop Subnet Manager.

   5. Use the setsmpriority 10 command to set the priority to 10.

   6. Use the enablesm command to restart Subnet Manager.

   7. Repeat the getmaster command to ensure the Subnet Manager Master is running on the spine switch.

2. Ensure the new rack is near the existing rack. The RDMA Network Fabric cables must be able to reach the servers in each rack.

3. Completely shut down the new rack (R2).

4. Cable the two leaf switches R2 IB2 and R2 IB3 in the new rack according to Two-Rack Cabling with InfiniBand Network Fabric. Note that you need to first remove the seven existing inter-switch connections between each leaf switch, as well as the two connections between the leaf switches and the spine switch in the new rack R2, not in the existing rack R1.

5. Verify both RDMA Network Fabric interfaces are up on all database nodes and storage cells. You can do this by running the ibstat command on each node and verifying both interfaces are up.

6. Power off leaf switch R1 IB2. This causes all the database servers and Exadata Storage Servers to fail over their RDMA Network Fabric traffic to R1 IB3.

7. Disconnect all seven inter-switch links between R1 IB2 and R1 IB3, as well as the one connection between R1 IB2 and the spine switch R1 IB1.

8. Cable leaf switch R1 IB2 according to Two-Rack Cabling with InfiniBand Network Fabric.

9. Power on leaf switch R1 IB2.

10. Wait for three minutes for R1 IB2 to become completely operational.

    To check the switch, log in to the switch and run the ibswitches command. The output should show three switches, R1 IB1, R1 IB2, and R1 IB3.

11. Verify both RDMA Network Fabric interfaces are up on all database nodes and storage cells. You can do this by running the ibstat command on each node and verifying both interfaces are up.

12. Power off leaf switch R1 IB3. This causes all the database servers and storage servers to fail over their RDMA Network Fabric traffic to R1 IB2.

13. Disconnect the one connection between R1 IB3 and the spine switch R1 IB1.

14. Cable leaf switch R1 IB3 according to Two-Rack Cabling with InfiniBand Network Fabric.

15. Power on leaf switch R1 IB3.

16. Wait for three minutes for R1 IB3 to become completely operational.

    To check the switch, log in to the switch and run the ibswitches command. The output should show three switches, R1 IB1, R1 IB2, and R1 IB3.

17. Power on all the InfiniBand switches in R2.

18. Wait for three minutes for the switches to become completely operational.

    To check the switch, log in to the switch and run the ibswitches command. The output should show six switches, R1 IB1, R1 IB2, R1 IB3, R2 IB1, R2 IB2, and R2 IB3.

19. Ensure the Subnet Manager Master is running on R1 IB1 by running the getmaster command from any switch.

20. Power on all servers in R2.

21. Log in to spine switch R1 IB1, and lower its priority to 8 as follows:

    1. Use the disablesm command to stop Subnet Manager.

    2. Use the setsmpriority 8 command to set the priority to 8.

    3. Use the enablesm command to restart Subnet Manager.

22. Ensure Subnet Manager Master is running on one of the spine switches.