8 Extending Recovery Appliance with Multiple Racks

This chapter describes how to cable together multiple Recovery Appliance racks. It contains the following sections:

See Also:

Multirack Cabling Tables

8.1 About Multirack Configurations

You can add racks together to increase system capacity and performance. Recovery Appliance has a flexible configuration. A rack has two compute servers and a minimum of three storage servers. A Recovery Appliance X6 or X5 full rack has 18 storage servers and a Recovery Appliance X4 full rack has 14 storage servers. The maximum configuration is 18 full racks.

You can cable together multiple Recovery Appliance racks that have any number of storage servers; they do not have to be full racks. Spine switches connect the InfiniBand network among multiple racks. You must order the spine switches and cables.

Before cabling the racks together:

  1. Determine the number of racks to be cabled together.

  2. Obtain the total number of InfiniBand cables required for the installation. See Multirack Cabling Tables.

  3. Ensure that each rack has a Sun Datacenter InfiniBand Switch 36 spine switch installed in the bottom rack unit (U1). See "Recovery Appliance X6 and X5 Rack Layout" or "Recovery Appliance X4 Rack Layout", as applicable.

  4. Gather the following tools:

    • Pen

    • Note pad

    • Flashlight

    • Labels for cables and rack

    • Cable ties (Oracle recommends hook and loop straps)

    See Also:

    "Parts for InfiniBand Switches" for information about cable labels

  5. Review the safety guidelines. See Zero Data Loss Recovery Appliance Safety and Compliance Guide.

8.2 Extending a Rack by Adding Another Rack

You can extend Recovery Appliance by adding it to another Recovery Appliance rack. The procedure consists of cabling and configuring the racks together. You can cable the racks with no downtime.

Note the following:

  • There is some performance degradation while cabling the racks together. This degradation results from reduced network bandwidth, and retransmitting data because of packet loss when a cable is unplugged.

  • The environment is not a high-availability environment because one leaf switch must be off. All traffic goes through the remaining leaf switch.

  • Only the existing rack is operational, and the new rack is powered down.

  • The software running on the systems must not have problems related to InfiniBand restarts.

  • The new racks must be configured with the appropriate IP addresses to be migrated into the expanded system before cabling. Duplicate IP addresses are not allowed.

  • The existing spine switch is set to priority 10 during the cabling procedure. This setting gives the spine switch a higher priority than any other switch in the fabric. The spine switch is the first to take the Subnet Manager Master role whenever a new Subnet Manager Master is set during the cabling procedure.

8.3 Adding the Spine Switches

A spine switch is required in each rack when configuring multiple racks on the same InfiniBand fabric. A spine switch can be factory installed before shipment, or it can be installed on site. Recovery Appliance uses a third Sun Datacenter InfiniBand Switch 36 as the spine switch.

Note:

This procedure is specific to Recovery Appliance. It differs from the steps in the Sun Datacenter InfiniBand Switch 36 manual.

  1. Review the guidelines for adding a rack component. See "Precautions and Guidelines".
  2. Unpack the Sun Datacenter InfiniBand Switch 36 switch components from the packing cartons:

    • Sun Datacenter InfiniBand Switch 36 switch

    • Cable bracket and rackmount kit

    • Cable management bracket and cover

    • 2 rack rail assemblies

    • Assortment of screws and captive nuts

    • Sun Datacenter InfiniBand Switch 36 documentation

    The service label procedure on top of the switch includes descriptions of the preceding items.

  3. Install cage nuts in each rack rail in the appropriate holes.
  4. Attach the brackets with cutouts to the power supply side of the switch.
  5. Attach the C-brackets to the switch on the side of the InfiniBand ports.
  6. Slide the switch halfway into the rack from the front. You must keep it to the left side of the rack as far as possible while pulling the two power cords through the C-bracket on the right side.
  7. Use a long-shaft, 16-inch, No. 3, Phillips screwdriver to install the slide rails from the rear of the rack into the C-brackets on the switch.
  8. Attach the cable arm brackets to the slide rail to the rear of the rack.
  9. Install the lower screw loosely, with the cable arm bracket rotated 90 degrees downward to allow access to the screw.
  10. Rotate the cable arm bracket to the correct position.
  11. Install the upper screw.
  12. Tighten both screws.
  13. Repeat steps 7 through 12 on the other side.
  14. Push the switch completely into the rack from the front, routing the power cords through the cutout on the rail bracket.
  15. Tighten the screws, using the Phillips screwdriver.
  16. Slide the server in rack location U2 out to the locked service position, to improve access to the switch.
  17. Install the lower part of the cable management arm across the back of the switch.
  18. Connect the cables to the appropriate ports.
  19. Install the upper part of the cable management arm.
  20. Slide the server in rack location U2 into the rack.
  21. Install power cords into the InfiniBand switch power supply slots on the front.
  22. Install the vented filler panel in front of the switch.

8.4 Floor and Underfloor Delivery of InfiniBand Cables

Sun Datacenter InfiniBand Switch 36 switch accepts InfiniBand cables from floor or underfloor delivery. Floor and underfloor delivery limits the tension in the InfiniBand cable to the weight of the cable for the rack height of the switch.

Note:

Overhead cabling details are not included in this guide. For details on overhead cabling, contact your Oracle service engineer.

8.5 Cabling Two Racks Together

This section describes how to cable together two adjacent racks. The existing rack is R1, and the new rack is R2.

To cable two racks together:

  1. Set the priority of the current, active Subnet Manager Master to 10 on the spine switch:

    1. Log in to any InfiniBand switch on the active system.

    2. Use the getmaster command to verify that the Subnet Manager Master is running on the spine switch. If it is not, then follow the procedure in "Setting a Spine Switch as the Subnet Manager Master".

    3. Log in to the spine switch.

    4. Use the disablesm command to stop Subnet Manager.

    5. Use the setsmpriority 10 command to set the priority to 10.

    6. Use the enablesm command to restart the Subnet Manager.

    7. Repeat step 1.b to ensure the Subnet Manager Master is running on the spine switch.

  2. Ensure the new rack is near the existing rack. The InfiniBand cables must be able to reach the servers in each rack.

  3. Completely shut down the new rack (R2).

  4. Cable the leaf switch in the new rack according to Table E-4.

  5. Power off leaf switch R1 IB2. This action causes all compute servers and storage servers to fail over their InfiniBand traffic to R1 IB3.

  6. Disconnect all interswitch links between R1IB2 and R1 IB3.

  7. Cable leaf switch R1 IB2 according to Table E-3.

  8. Power on leaf switch R1 IB2.

  9. Wait three minutes for R1 IB2 to become completely operational.

    To check the switch, log in to it and run the ibswitches command. The output should show three switches, R1 IB1, R1 IB2, and R1 IB3.

  10. Power off leaf switch R1 IB3. This action causes all compute servers and storage servers to fail over their InfiniBand traffic to R1 IB2.

  11. Cable leaf switch R1 IB3 according to Table E-3.

  12. Power on leaf switch R1 IB3.

  13. Wait three minutes for R1 IB3 to become completely operational.

  14. To check the switch, log in to it and run the ibswitches command. The output should show three switches, R1 IB1, R1 IB2, and R1 IB3.

  15. Power on all InfiniBand switches in R2.

  16. Wait for three minutes for the switches to become completely operational.

  17. To check the switch, log in to it and run the ibswitches command. The output should show six switches, R1 IB1, R1 IB2, R1 IB3, R2 IB1, R2 IB2, and R2 IB3.

  18. Run the getmaster command from any switch to ensure that the Subnet Manager Master is running on R1 IB1.

  19. Power on all servers in R2.

  20. Log in to spine switch R1 IB1, and lower its priority to 8:

    1. Use the disablesm command to stop the Subnet Manager.

    2. Use the setsmpriority 8 command to set the priority to 8.

    3. Use the enablesm command to restart the Subnet Manager.

  21. Ensure that the Subnet Manager Master is running on one of the spine switches.

After cabling the racks together, configure them as described in Configuring a Recovery Appliance Rack.

8.6 Cabling Several Racks Together

This section describes how to cable several adjacent racks together. The existing racks are R1, R2,... Rn, the new rack is Rn+1, and the Subnet Manager Master is running on R1 IB1.

To cable several racks together:

  1. Set the priority of the current, active Subnet Manager Master to 10 on the spine switch:

    1. Log in to any InfiniBand switch on the active system.

    2. Use the getmaster command to verify that the Subnet Manager Master is running on the spine switch. If it is not, then follow the procedure in "Setting a Spine Switch as the Subnet Manager Master".

    3. Log in to the spine switch.

    4. Use the disablesm command to stop the Subnet Manager.

    5. Use the setsmpriority 10 command to set the priority to 10.

    6. Use the enablesm command to restart the Subnet Manager.

    7. Repeat step 1.b to ensure that the Subnet Manager Master is running on the spine switch.

  2. Ensure that the new rack is near the existing rack. The InfiniBand cables must be able to reach the servers in each rack.

  3. Completely shut down the new rack (Rn+1).

  4. Cable the leaf switch in the new rack according to the appropriate table in Multirack Cabling Tables. For example, if rack Rn+1 is R4, then use Table E-11.

  5. Complete the following procedure for each of the original racks. In these steps, Rx represents a rack number from R1 to Rn.

    1. Power off leaf switch Rx IB2. This action causes all compute servers and storage servers to fail over their InfiniBand traffic to Rx IB3.

    2. Cable leaf switch Rx IB2 according to Multirack Cabling Tables.

    3. Power on leaf switch Rx IB2.

    4. Wait three minutes for Rx IB2 to become completely operational.

    5. To check the switch, log in to it and run the ibswitches command. The output should show n*3 switches for IB1, IB2, and IB3 in racks R1, R2,... Rn.

    6. Power off leaf switch Rx IB3. This action causes all compute servers and storage servers to fail over their InfiniBand traffic to Rx IB2.

    7. Cable leaf switch Rx IB3 according to Multirack Cabling Tables.

    8. Power on leaf switch Rx IB3.

    9. Wait three minutes for Rx IB3 to become completely operational.

    10. To check the switch, log in to the switch and run the ibswitches command. The output should show n*3 switches for IB1, IB2, and IB3 in racks R1, R2,... Rn.

    11. Repeat these steps for each rack until all of them are recabled according to Multirack Cabling Tables.

  6. Power on all InfiniBand switches in Rn+1.

  7. Wait three minutes for the switches to become completely operational.

  8. To check the switch, log in to it and run the ibswitches command. The output should show (n+1)*3 switches for IB1, IB2, and IB3 in racks R1, R2,... Rn+1.

  9. Run the getmaster command from any switch to ensure that the Subnet Manager Master is running on R1 IB1.

  10. Power on all servers in Rn+1.

  11. Log in to spine switch R1 IB1, and lower its priority to 8:

    1. Use the disablesm command to stop the Subnet Manager.

    2. Use the setsmpriority 8 command to set the priority to 8.

    3. Use the enablesm command to restart the Subnet Manager.

  12. To ensure that the Subnet Manager Master is running on one of the spine switches, use the getmaster command from any switch.

  13. To ensure that the Subnet Manager is running on every spine switch, use the following command from any switch:

    ibdiagnet -r

    The Summary Fabric SM-state-priority section of the output should show that the spine switches are running.

  14. If a spine switch is not running, then log in to the switch and use the enablesm command to enable the Subnet Manager.

  15. If there are now four or more racks, then log in to the leaf switches in each rack and use the disablesm command to disable the Subnet Manager

8.7 Configuring Multiple Racks

In addition to the hardware installation of the new racks, you must do the following procedures: