4.3 Two-Rack Cabling for X9M and Later Model Racks
This section provides the cabling details for X9M and later models to connect two racks, both of which use RoCE Network Fabric.
Note:
-
The following conventions are used in the cabling notation for connecting multiple racks together:
-
The abbreviation for the first rack is R1, the second rack is R2, and so on.
-
LL identifies a lower leaf switch and UL identifies an upper leaf switch.
-
SS identifies the spine switch, which is located in U1 on all racks.
-
A specific switch is identified by combining abbreviations. For example, R1LL identifies the lower leaf switch (LL) on the first rack (R1).
-
-
The leaf switches are located as follows:
-
At rack unit 20 (U20) and 22 (U22) in 2-socket systems (Oracle Exadata Rack X9M-2 and later models).
-
At rack unit 21 (U21) and rack unit 23 (U23) in 8-socket systems (Oracle Exadata X9M-8).
-
-
The cable lengths shown in the following lists assume that the racks are adjacent to each other, the cables are routed through a raised floor, and there are no obstacles in the routing between the racks. If the racks are not adjacent, or use overhead cabling trays, then they may require longer cable lengths. Cable lengths up to 100 meters are supported.
-
Only optical cables (with additional transceivers) are supported for lengths greater than 5 meters.
-
For X9M-8 systems with three database servers and 11 storage servers only, port 30 on the leaf switches is connected to a database server and is not used as an inter-switch link. Consequently, for these systems only, ignore the connections to port number 30 on every leaf switch in the following tables. This adjustment leaves only 13 inter-switch links on each leaf switch and only applies to X9M-8 systems with three database servers and 11 storage servers.
The following tables contain details for all of the RoCE Network Fabric cabling connections in a two-rack system.
Table 4-1 Leaf Switch Connections for the First Rack in a Two-Rack System
| Switch Pair | Connections | Cable Length |
|---|---|---|
|
R1UL to R1SS |
R1UL-P1 to R1SS-P21 R1UL-P2 to R1SS-P23 R1UL-P3 to R1SS-P25 R1UL-P4 to R1SS-P9 R1UL-P5 to R1SS-P5 R1UL-P6 to R1SS-P11 R1UL-P7 to R1SS-P7 |
3 meters |
|
R1UL to R2SS |
R1UL-P30 to R2SS-P9 R1UL-P31 to R2SS-P5 R1UL-P32 to R2SS-P11 R1UL-P33 to R2SS-P7 R1UL-P34 to R2SS-P21 R1UL-P35 to R2SS-P23 R1UL-P36 to R2SS-P25 |
5 meters |
|
R1LL to R1SS |
R1LL-P1 to R1SS-P27 R1LL-P2 to R1SS-P29 R1LL-P3 to R1SS-P31 R1LL-P4 to R1SS-P17 R1LL-P5 to R1SS-P13 R1LL-P6 to R1SS-P19 R1LL-P7 to R1SS-P15 |
3 meters |
|
R1LL to R2SS |
R1LL-P30 to R2SS-P17 R1LL-P31 to R2SS-P13 R1LL-P32 to R2SS-P19 R1LL-P33 to R2SS-P15 R1LL-P34 to R2SS-P27 R1LL-P35 to R2SS-P29 R1LL-P36 to R2SS-P31 |
5 meters |
Table 4-2 Leaf Switch Connections for the Second Rack in a Two-Rack System
| Switch Pair | Connections | Cable Length |
|---|---|---|
|
R2UL to R1SS |
R2UL-P1 to R1SS-P22 R2UL-P2 to R1SS-P24 R2UL-P3 to R1SS-P26 R2UL-P4 to R1SS-P10 R2UL-P5 to R1SS-P6 R2UL-P6 to R1SS-P12 R2UL-P7 to R1SS-P8 |
5 meters |
|
R2UL to R2SS |
R2UL-P30 to R2SS-P10 R2UL-P31 to R2SS-P6 R2UL-P32 to R2SS-P12 R2UL-P33 to R2SS-P8 R2UL-P34 to R2SS-P22 R2UL-P35 to R2SS-P24 R2UL-P36 to R2SS-P26 |
3 meters |
|
R2LL to R1SS |
R2LL-P1 to R1SS-P28 R2LL-P2 to R1SS-P30 R2LL-P3 to R1SS-P32 R2LL-P4 to R1SS-P18 R2LL-P5 to R1SS-P14 R2LL-P6 to R1SS-P20 R2LL-P7 to R1SS-P16 |
5 meters |
|
R2LL to R2SS |
R2LL-P30 to R2SS-P18 R2LL-P31 to R2SS-P14 R2LL-P32 to R2SS-P20 R2LL-P33 to R2SS-P16 R2LL-P34 to R2SS-P28 R2LL-P35 to R2SS-P30 R2LL-P36 to R2SS-P32 |
3 meters |
The following table contains all of the RoCE Network Fabric cabling connections from the previous tables. In this table, the connections are sorted by the spine switch port location.
Table 4-3 Two-Rack System Connections Sorted By The Spine Switch Port Location
| Spine Switch Port | Leaf Switch Port | Cable Length |
|---|---|---|
| R1SS-P5 | R1UL-P5 | 3 meters |
| R1SS-P6 | R2UL-P5 | 5 meters |
| R1SS-P7 | R1UL-P7 | 3 meters |
| R1SS-P8 | R2UL-P7 | 5 meters |
| R1SS-P9 | R1UL-P4 | 3 meters |
| R1SS-P10 | R2UL-P4 | 5 meters |
| R1SS-P11 | R1UL-P6 | 3 meters |
| R1SS-P12 | R2UL-P6 | 5 meters |
| R1SS-P13 | R1LL-P5 | 3 meters |
| R1SS-P14 | R2LL-P5 | 5 meters |
| R1SS-P15 | R1LL-P7 | 3 meters |
| R1SS-P16 | R2LL-P7 | 5 meters |
| R1SS-P17 | R1LL-P4 | 3 meters |
| R1SS-P18 | R2LL-P4 | 5 meters |
| R1SS-P19 | R1LL-P6 | 3 meters |
| R1SS-P20 | R2LL-P6 | 5 meters |
| R1SS-P21 | R1UL-P1 | 3 meters |
| R1SS-P22 | R2UL-P1 | 5 meters |
| R1SS-P23 | R1UL-P2 | 3 meters |
| R1SS-P24 | R2UL-P2 | 5 meters |
| R1SS-P25 | R1UL-P3 | 3 meters |
| R1SS-P26 | R2UL-P3 | 5 meters |
| R1SS-P27 | R1LL-P1 | 3 meters |
| R1SS-P28 | R2LL-P1 | 5 meters |
| R1SS-P29 | R1LL-P2 | 3 meters |
| R1SS-P30 | R2LL-P2 | 5 meters |
| R1SS-P31 | R1LL-P3 | 3 meters |
| R1SS-P32 | R2LL-P3 | 5 meters |
| R2SS-P5 | R1UL-P31 | 5 meters |
| R2SS-P6 | R2UL-P31 | 3 meters |
| R2SS-P7 | R1UL-P33 | 5 meters |
| R2SS-P8 | R2UL-P33 | 3 meters |
| R2SS-P9 | R1UL-P30 | 5 meters |
| R2SS-P10 | R2UL-P30 | 3 meters |
| R2SS-P11 | R1UL-P32 | 5 meters |
| R2SS-P12 | R2UL-P32 | 3 meters |
| R2SS-P13 | R1LL-P31 | 5 meters |
| R2SS-P14 | R2LL-P31 | 3 meters |
| R2SS-P15 | R1LL-P33 | 5 meters |
| R2SS-P16 | R2LL-P33 | 3 meters |
| R2SS-P17 | R1LL-P30 | 5 meters |
| R2SS-P18 | R2LL-P30 | 3 meters |
| R2SS-P19 | R1LL-P32 | 5 meters |
| R2SS-P20 | R2LL-P32 | 3 meters |
| R2SS-P21 | R1UL-P34 | 5 meters |
| R2SS-P22 | R2UL-P34 | 3 meters |
| R2SS-P23 | R1UL-P35 | 5 meters |
| R2SS-P24 | R2UL-P35 | 3 meters |
| R2SS-P25 | R1UL-P36 | 5 meters |
| R2SS-P26 | R2UL-P36 | 3 meters |
| R2SS-P27 | R1LL-P34 | 5 meters |
| R2SS-P28 | R2LL-P34 | 3 meters |
| R2SS-P29 | R1LL-P35 | 5 meters |
| R2SS-P30 | R2LL-P35 | 3 meters |
| R2SS-P31 | R1LL-P36 | 5 meters |
| R2SS-P32 | R2LL-P36 | 3 meters |