4.2 Preparing for Multi-Rack Cabling with X9M and Later Model Racks

Racks can be added together to increase system capacity and performance. When cabling racks together, note the following:

• The cable lengths shown in this document assume the racks are adjacent to each other. If the racks are not adjacent, or there are obstacles in the raised floor, or if you use overhead cabling, then longer cables may be required. For optical cables, the maximum supported cable length is 100 meters. For copper cables, the maximum supported cable length is 5 meters.

• Oracle recommends that the names for the servers include the rack unit number. This helps identify the server during diagnostics.

• When completing Oracle Exadata Deployment Assistant (OEDA) for the additional rack, you are prompted for SCAN addresses. However, these SCAN addresses are not used because the SCAN address from the original rack are used. Manually remove the new SCAN addresses from the generated installation files.

• The software owner account names and group names, as well as their identifiers, must match the names and identifiers of the original rack.

• If the additional grid disks are used with existing disk groups, then ensure the grid disk sizes for the new rack are the same as the original rack.

• For multi-rack configurations containing up to 14 racks, a spine switch must exist in each rack in order to interconnect the RoCE Network Fabric.

Perform the following tasks before cabling racks together:

1. Determine the number of racks that will be cabled together.

2. Order the parts needed to connect the racks.

   To extend Oracle Exadata racks with RoCE Network Fabric, for each rack being added you must order extra cables, transceivers for longer cables, and a RoCE Network Fabric spine switch, if one is required.

   When connecting four or more racks, or if you need longer cables for your environment, you must purchase additional 10 meter or 15 meter fiber cables with two QSFP28 SR transceivers to connect each end. The QSFP28 SR transceivers are needed for fiber cables over 5 meters in length.

   For multi-rack configurations containing up to 14 racks, the following table outlines the cables needed to interconnect the racks:

   Number of Racks	3 Meter Cables Required	5 Meter Cables Required	10 Meter Cables Required	15 Meter Cables Required
   2	28	28	0	0
   3	28	56	0	0
   4	28	71	13	0
   5	28	78	34	0
   6	28	87	53	0
   7	28	88	80	0
   8	28	91	105	0
   9	28	94	130	0
   10	28	95	151	6
   11	28	97	168	15
   12	28	99	184	25
   13	28	100	196	40
   14	28	100	204	60

3. Gather the following tools:

   • Pen
   • Note pad
   • Flashlight
   • Labels for cables and rack
   • Cable ties (Oracle recommends hook and loop straps)

4. Determine a naming method for the rack prefixes. For example, if the original rack has the prefix dbm01, then use the prefix dbm02 for the second rack, the prefix dbm03 for the third rack, and so on.

5. Verify that the racks have unique host names and IP addresses. All servers interconnected in the racks must have unique names and IP addresses.

   Server name and IP address conventions may differ in the following cases:

   • Initial installation of all Oracle Exadata Racks: System address assignments and host names should be complete.

   • New Oracle Exadata Racks are added to an existing cluster: The new rack configuration should require unique host names and IP addresses for the new Oracle Exadata. The IP addresses on the same subnet cannot conflict with the existing systems.

   • Two existing Oracle Exadata Racks are clustered together: You can assign host names and IP addresses only if Oracle Exadata racks are already assigned unique host names and IP addresses, or the entire cluster must be reconfigured. The machines must be on the same subnet and not have conflicting IP addresses.

6. Ensure the IP addresses for the new servers are in the same subnet, and do not overlap with the currently-installed servers.

7. Check that the firmware on the original switches is at the same level as the new switches by using the sh ver command.

   It is highly recommended, though not mandatory, to use the same firmware version on all of the switches. If the firmware is not at the same level, you can apply a firmware patch to bring the switches up to the same firmware level.

The remaining topics in this section provide detailed cabling information for cabling up to 14 racks together.