1. Owner's Guide
  2. Hardware Installation and Configuration
  3. Connecting Multiple Oracle Big Data Appliance Racks

9 Connecting Multiple Oracle Big Data Appliance Racks

This chapter describes how to combine multiple Oracle Big Data Appliance racks into one large cluster. It contains the following sections:

See Also:

Multirack Cabling Tables

9.1 Extending a Rack by Adding Another Rack

When creating a multirack Hadoop cluster or providing access to Oracle Big Data Appliance from an Oracle Exadata Database Machine, you must connect multiple racks to each other. Racks can be cabled together with no downtime.

During the cabling procedure, note the following:

  • There is some performance degradation while you are cabling the racks together. This degradation results from reduced network bandwidth, and the data retransmission due to packet loss when a cable is unplugged.

  • The environment is not a high-availability environment because one leaf switch must be off. All traffic goes through the remaining leaf switch.

  • Only the existing rack is operational, and any new rack is powered down.

  • The software running on the systems must not have problems related to InfiniBand restarts.

  • The new racks must be configured with the appropriate IP addresses to be migrated into the expanded system before any cabling, and duplicate IP addresses are not allowed.

  • The existing spine switch is set to priority 10 during the cabling procedure. This setting gives the spine switch a higher priority than any other switch in the network fabric. The spine switch is first to take the Subnet Manager Master role whenever a new Subnet Manager Master is set during the cabling procedure.

  • To extend a cluster to a new rack, there must be at least four nodes in the first rack. The minimum for racks subsequently added to the cluster is three nodes.

See Also:

9.2 Prerequisites for Multirack Cabling

Check the Firmware on all Switches

Verify that all Sun Network QDR InfiniBand Gateway switches on all connected Oracle Big Data Appliance and Oracle Exadata Database Machine racks are running the supported firmware. The minimum is version NM2 2.1.8-2. Version 2.2.9-3 is recommended. Update the firmware as necessary.

Note:

NM2 2.2.7-2 is a prerequisite for further upgrades. Before upgrading to either the 2.2.8-2 or 2.2.9-3 firmware, you must first upgrade to 2.2.7-2. Otherwise, the firmware update will fail.

When updating the firmware of Sun Network QDR InfiniBand Gateway switches from v2.0.x to v2.1.x or higher, be careful to update them in descending order of the gateway instance numbers. Start with the switch with the highest gateway instance number.

From the Oracle ILOM interface, use this hardware command to obtain the gateway instance number:

setgwinstance --list

See My Oracle Support Doc 1534255.1 for instructions on upgrading the InfiniBand firmware on Oracle Big Data Appliance.

Use Oracle-Supplied Cables

Customers are advised to use only Oracle-supplied cables for connections to the InfiniBand switches, including connections to Exadata Database Machine or to other Oracle-supplied engineered systems. Oracle does not certify or validate cables from other sources and cannot provide support for any issues that may arise from use of third-party cables for these rack-to-rack connections.

9.3 Cabling Two Racks Together

The following procedure describes how to cable two racks together. This procedure assumes that the racks are adjacent to each other. In the procedure, the existing rack is R1, and the new rack is R2.

To cable two racks together:

  1. Set the priority of the current, active Subnet Manager Master to 10 on the spine switch, as follows:

    1. Log in to any InfiniBand switch on the active system.

    2. Use the getmaster command to verify that the Subnet Manager Master is running on the spine switch.

    3. Log in to the spine switch.

    4. Use the disablesm command to stop the Subnet Manager.

    5. Use the setsmpriority 10 command to set the priority to 10.

    6. Use the enablesm command to restart the Subnet Manager.

    7. Repeat Step 1.b to ensure that the Subnet Manager Master is running on the spine switch.

  2. Ensure that the new rack is near the existing rack. The InfiniBand cables must be able to reach the servers in each rack.

  3. Completely shut down the new rack (R2).

  4. Cable the leaf switch in the new rack according to Table F-2.

  5. Shut down leaf switch R1 IB2. This causes all servers to fail over their InfiniBand traffic to R1 IB3.

  6. Disconnect all interswitch links between R1 IB2 and R1 IB3.

  7. Cable leaf switch R1 IB2 according to Table F-1.

  8. Power on leaf switch R1 IB2.

  9. Wait for 3 minutes for R1 IB2 to become completely operational.

    To check the switch, log in to it and run the ibswitches command. The output should show three switches: R1 IB1, R1 IB2, and R1 IB3.

  10. Shut down leaf switch R1 IB3. This causes all servers to fail over their InfiniBand traffic to R1 IB2.

  11. Cable leaf switch R1 IB3 according to Table F-1.

  12. Power on leaf switch R1 IB3.

  13. Wait for 3 minutes for R1 IB3 to become completely operational.

    To check the switch, log in to it and run the ibswitches command. The output should show three switches: R1 IB1, R1 IB2, and R1 IB3.

  14. Power on all the InfiniBand switches in R2.

  15. Wait for 3 minutes for the switches to become completely operational.

    To check the switch, log in to it and run the ibswitches command. The output should show six switches: R1 IB1, R1 IB2, R1 IB3, R2 IB1, R2 IB2, and R2 IB3.

  16. Ensure that the Subnet Manager Master is running on R1 IB1 by running the getmaster command from any switch.

  17. Power on all servers in R2.

  18. Log in to spine switch R1 IB1, and lower its priority to 8 as follows:

    1. Use the disablesm command to stop the Subnet Manager.

    2. Use the setsmpriority 8 command to set the priority to 8.

    3. Use the enablesm command to restart the Subnet Manager.

  19. Ensure that the Subnet Manager Master is running on one of the spine switches.

After cabling the racks together, proceed to configure the racks.

9.4 Cabling Several Racks Together

The following procedure describes how to cable several racks together. This procedure assumes that the racks are adjacent to each other. In the procedure, the existing racks are R1, R2,... Rn, the new rack is Rn+1, and the Subnet Manager Master is running on R1 IB1.

You can cable up to 18 racks together without additional switches.

To cable several racks together:

  1. Set the priority of the current, active Subnet Manager Master to 10 on the spine switch, as follows:

    1. Log in to any InfiniBand switch on the active system.

    2. Use the getmaster command to verify that the Subnet Manager Master is running on the spine switch.

    3. Log in to the spine switch.

    4. Use the disablesm command to stop the Subnet Manager.

    5. Use the setsmpriority 10 command to set the priority to 10.

    6. Use the enablesm command to restart the Subnet Manager.

    7. Repeat Step 1.b to ensure that the Subnet Manager Master is running on the spine switch.

  2. Ensure that the new rack is near the existing rack. The InfiniBand cables must be able to reach the servers in each rack.

  3. Completely shut down the new rack (Rn+1).

  4. Cable the leaf switch in the new rack according to the appropriate table in Multirack Cabling Tables. For example, if rack Rn+1 was R4, then use Table F-9.

    Caution:

    Cabling within a live network must be done carefully in order to avoid potentially serious disruptions.

    The cabling table that you use for your new InfiniBand topology tells you how to connect ports on the leaf switches to ports on spine switches in order to connect the racks. Some of these ports on the spine switches might be already in use to support the existing InfiniBand topology. In these cases, connect only the cable on the leaf switch in the new rack and stop there for now. Make note of which cables you were not able to terminate.

    Do not unplug any cables on the spine switch in the existing rack at this point. Step 5 describes how to re-cable the leaf switches on the existing racks (one leaf switch after the other - while the leaf switch being re-cabled will be powered off), which will free up these currently in-use ports. At that point, you can connect the other end of the cable from the leaf switch in the new rack to the spine switch in the existing rack as indicated in the table.

    .

  5. Complete the following procedure for each of the original racks. In these steps, Rx represents a rack number from R1 to Rn.

    1. Shut down leaf switch Rx IB2. This causes all servers to fail over their InfiniBand traffic to Rx IB3.

    2. Cable leaf switch Rx IB2 according to Multirack Cabling Tables.

    3. Power on leaf switch Rx IB2.

    4. Wait for 3 minutes for Rx IB2 to become completely operational.

      To check the switch, log in it and run the ibswitches command. The output should show n*3 switches for IB1, IB2, and IB3 in racks R1, R2,... Rn.

    5. Shut down leaf switch Rx IB3. This causes all servers to fail over their InfiniBand traffic to Rx IB2.

    6. Cable leaf switch Rx IB3 according to Multirack Cabling Tables.

    7. Power on leaf switch Rx IB3.

    8. Wait for 3 minutes for Rx IB3 to become completely operational.

      To check the switch, log in to the switch and enter the ibswitches command. The output should show n*3 switches for IB1, IB2, and IB3 in racks R1, R2,... Rn.

      All racks should now be rewired according to Multirack Cabling Tables.

  6. Power on all the InfiniBand switches in Rn+1.

  7. Wait for 3 minutes for the switches to become completely operational.

    To check the switch, log in to the switch and run the ibswitches command. The output should show (n+1)*3 switches for IB1, IB2, and IB3 in racks R1, R2,... Rn+1.

  8. Ensure that the Subnet Manager Master is running on R1 IB1 by entering the getmaster command from any switch.

  9. Power on all servers in Rn+1.

  10. Log in to spine switch R1 IB1, and lower its priority to 8 as follows:

    1. Enter the disablesm command to stop the Subnet Manager.

    2. Enter the setsmpriority 8 command to set the priority to 8.

    3. Enter the enablesm command to restart the Subnet Manager.

  11. Ensure that the Subnet Manager Master is running on one of the spine switches by entering the getmaster command from any switch.

  12. Ensure that the Subnet Manager is running on every spine switch by entering the following command from any switch:

    ibdiagnet -r

    Each spine switch should show as running in the Summary Fabric SM-state-priority section of the output. If a spine switch is not running, then log in to the switch and enable the Subnet Manager by entering the enablesm command.

  13. If there are now four or more racks, then log in to the leaf switches in each rack and disable the Subnet Manager by entering the disablesm command.