5.1.1 Preparing for Multi-Rack Cabling with X8M Racks

Racks can be added together to increase system capacity and performance. When cabling racks together, note the following:

• The cable lengths shown in this document assume the racks are adjacent to each other. If the racks are not adjacent, or there are obstacles in the raised floor, or if you use overhead cabling, then longer cables may be required. For optical cables, the maximum supported cable length is 100 meters. For copper cables, the maximum supported cable length is 5 meters.

• Oracle recommends that the names for the servers include the rack unit number. This helps identify the server during diagnostics.

• When completing Oracle Exadata Deployment Assistant (OEDA) for the additional rack, you are prompted for SCAN addresses. However, these SCAN addresses are not used because the SCAN address from the original rack are used. Manually remove the new SCAN addresses from the generated installation files.

• The software owner account names and group names, as well as their identifiers, must match the names and identifiers of the original rack.

• If the additional grid disks are used with existing disk groups, then ensure the grid disk sizes for the new rack are the same as the original rack.

Note:

Oracle Exadata Eighth Racks, Quarter Racks, and Elastic Configurations are connected to other racks in the same fashion as larger racks are connected to each other. In other words, a spine switch must exist in each rack in order to interconnect with other racks.

Perform the following tasks before cabling racks together:

1. Determine the number of racks that will be cabled together.

2. Order the parts needed to connect the racks.

   To extend Oracle Exadata racks with RoCE Network Fabric, for each rack being added you must order extra cables, transceivers for longer cables, and a RoCE Network Fabric spine switch, if one is not already available.

   When connecting four or more racks, or if you need longer cables for your environment, you must purchase additional 10 meter fiber cables with two QSFP28 SR transceivers to connect each end. The QSFP28 SR transceivers are needed for fiber cables over 5 meters in length.

   The following table lists the number of racks, and cables needed to interconnect up to eight racks:

   Number of Racks	3 Meter Cables Needed	5 Meter Cables Needed	10 Meter Cables Needed
   2	16	16	0
   3	16	32	0
   4	16	40	8
   5	16	46	18
   6	17	49	30
   7	19	51	42
   8	16	52	60

3. Gather the following tools:

   • Pen
   • Note pad
   • Flashlight
   • Labels for cables and rack
   • Cable ties (Oracle recommends hook and loop straps)

4. Determine a naming method for the rack prefixes. For example, if the original rack has the prefix dbm01, then use the prefix dbm02 for the second rack, the prefix dbm03 for the third rack, and so on.

5. Verify the racks have unique host names and IP addresses. All servers interconnected in the racks must have unique names and IP addresses.

   Server names and IP addresses conventions may differ in the following cases:

   • Initial installation of all Oracle Exadata Racks: System address assignments and host names should be complete.

   • New Oracle Exadata Racks are added to an existing cluster: The new rack configuration should require unique host names and IP addresses for the new Oracle Exadata. The IP addresses on the same subnet cannot conflict with the existing systems.

   • Two existing Oracle Exadata Racks are clustered together: You can assign host names and IP addresses only if Oracle Exadata racks are already assigned unique host names and IP addresses, or the entire cluster must be reconfigured. The machines must be on the same subnet and not have conflicting IP addresses.

6. Ensure the IP addresses for the new servers are in the same subnet, and do not overlap with the currently-installed servers.

7. Check that the firmware on the original switches is at the same level as the new switches by using the sh ver command.

   It is highly recommended, though not mandatory, to use the same firmware version on all of the switches. If the firmware is not at the same level, you can apply a firmware patch to bring the switches up to the same firmware level.