Preparing for Multi-Rack Cabling with InfiniBand Network Fabric

Racks can be added together to increase system capacity and performance. When cabling racks together, note the following:

• The cable lengths shown in this document assume the racks are adjacent to each other. If the racks are not adjacent, or there are obstacles in the raised floor, or if you use overhead cabling, then longer cables may be required. For optical cables, the maximum supported cable length is 100 meters. For copper cables, the maximum supported cable length is 5 meters.

• Oracle recommends that the names for the servers include the rack unit number. This helps identify the server during diagnostics.

• When completing Oracle Exadata Deployment Assistant for the additional rack, you are prompted for SCAN addresses. However, these SCAN addresses are not used because the SCAN address from the original rack are used. Manually remove the new SCAN addresses from the generated installation files.

• The software owner account names and group names, as well as their identifiers, must match the names and identifiers of the original rack.

• If the additional grid disks are used with existing disk groups, then ensure the grid disk sizes for the new rack are the same as the original rack.

• If the InfiniBand network consists of four or more racks cabled together, then disable the Subnet Manager on the leaf switches.

• Verify the Master Subnet Manager is located on the spine switch.

• Oracle Exadata Database Machine Quarter Racks can be extended as follows:

  • Connect two Oracle Exadata Database Machine Quarter Racks together. At least four of the six ports reserved for external connectivity are open on each leaf switch. The six ports are 5B, 6A, 6B, 7A, 7B, and 12A in each leaf switch. Maintain the existing seven inter-switch links between the leaf switches within each rack. Connect the leaf switches between the racks with two links each, using the ports reserved for external connectivity.

  • Connect one Oracle Exadata Database Machine Quarter Rack with one Oracle Exadata Database Machine Half Rack or one Oracle Exadata Database Machine Full Rack. At least four ports reserved for external connectivity are open on each leaf switch. The spine switch in the Oracle Exadata Database Machine Half Rack or Oracle Exadata Database Machine Full Rack remains as the spine switch. Maintain the existing seven inter-switch links between the leaf switches within each rack. Connect the leaf switches between the racks with two links each, using the ports reserved for external connectivity.

  • Connect one Oracle Exadata Database Machine Quarter Rack with two or more Oracle Exadata Database Machine Half Racks or Oracle Exadata Database Machine Full Racks. The racks are interconnected using a fat-tree topology. Connect each leaf switch in the quarter rack to the spine switch of each half rack or full rack using two links each. If there are more than four racks, then use one link instead of two. The seven inter-switch links between the leaf switches in the quarter rack are removed.

    Note: To connect more than one quarter rack to additional racks, it is necessary to purchase Sun Datacenter InfiniBand Switch 36 switches for the quarter racks.

Perform the following tasks before cabling racks together:

1. Determine the number of racks that will be cabled together.

2. Count the spare cables from the kit, and existing inter-switch cables.

   When connecting four or more racks, it is necessary to purchase additional 10 meter optical QSFP-to-QSFP QDR InfiniBand cables. Two QSFP SR transceivers are needed for each optical cable. The following table lists the number of racks, and cables needed to interconnect the racks.

   Number of Racks	3 Meter Cables Needed	5 Meter Cables Needed	10 Meter Cables Needed
   2	16	16	0
   3	18	30	0
   4	16	40	8
   5	20	42	18
   6	24	46	26
   7	28	44	40
   8	16	52	60

3. Gather the following tools:

   • Pen

   • Note pad

   • Flashlight

   • Labels for cables and rack

   • Cable ties (Oracle recommends hook and loop straps)

4. Determine a naming method for the rack prefixes. For example, if the original rack has the prefix dbm01, then use the prefix dbm02 for the second rack, the prefix dbm03 for the third rack, and so on.

5. Verify the racks have unique host names and IP addresses. All servers interconnected in the racks must have unique names and IP addresses.

   Server names and IP addresses conventions may differ in the following cases:

   • Initial installation of all Oracle Exadata Database Machine Full Racks: System address assignments and host names should be complete.

   • New Oracle Exadata Database Machine Full Racks are added to an existing cluster: The new rack configuration should require unique host names and IP addresses for the new Oracle Exadata Database Machines. The IP addresses on the same subnet cannot conflict with the existing systems.

   • Two existing Oracle Exadata Database Machine Full Racks are clustered together: You can assign host names and IP addresses only if Oracle Exadata Database Machines are already assigned unique host names and IP addresses, or the entire cluster must be reconfigured. The machines must be on the same subnet and not have conflicting IP addresses.

6. Ensure the IP addresses for the new servers are in the same subnet, and do not overlap with the currently-installed servers.

7. Ensure the firmware on the original switches are at the same level as the new switches using the nm2version command. If the firmware is not at the same level, then apply a firmware patch.