5.1 Understanding Multi-Rack Cabling for X8M Racks
Up to eight racks can be cabled together without external RDMA Network Fabric switches.
When cabling together multiple racks, each rack must contain three RDMA Network Fabric switches.
Two of the switches, known as leaf switches, connect to all of the database servers and storage servers in the rack. The leaf switches are located as follows:
-
At rack unit 20 (U20) and 22 (U22) in Oracle Exadata X8M-2.
-
At rack unit 21 (U21) and 23 (U23) in Oracle Exadata X8M-8.
The third switch, known as the spine switch, interconnects the leaf switches in a multi-rack configuration. The spine switch is located at rack unit 1 (U1). If any rack does not include a spine switch, then order the expansion switch kit.
The procedures in this section assume the racks are adjacent to each other, standard routing in raised floor is used, and there are no obstacles in the raised floor. If these assumptions are not correct for your environment, then longer cables may be required for the connections.
Note:
By default, Oracle Exadata Database Machine X8M racks do not include spare cables or a third RoCE Network Fabric switch. To extend these racks, you must order the required cables and RoCE Network Fabric switch.The following diagram shows the default RDMA Network Fabric architecture for a single-rack system. Each rack has two leaf switches, with eight connections between the leaf switches. The database servers and storage servers are each connected to both leaf switches. Each server contains a dual-port RDMA Network Fabric card, with port 1 connected to the lower leaf switch and port 2 connected to the upper leaf switch.
When connecting up to eight racks together, remove the eight existing inter-switch connections between each leaf switch on each rack. From each leaf switch, distribute eight connections over the spine switches in all racks. In multi-rack environments, the leaf switches inside a rack are no longer directly interconnected, as shown in the following graphic:
As shown in the preceding graphic, each leaf switch in rack 1 has the following connections:
- Four connections to its internal spine switch
- Four connections to the spine switch in rack 2
The spine switch in rack 1 has the following connections:
- Four connections to each leaf switch in rack 1
- Four connections to each leaf switch in rack 2
As the number of racks increases from two to eight, the pattern continues as shown in the following graphic:
Figure 5-1 Connections Between Spine Switches and Leaf Switches for up to 8 Racks
As shown in the preceding graphic, each leaf switch has eight inter-switch connections distributed over all spine switches. Each spine switch has 16 inter-switch connections distributed over all leaf switches. The leaf switches are not directly interconnected with other leaf switches, and the spine switches are not directly interconnected with the other spine switches.