4.5 Four-Rack Cabling for X9M and Later Model Racks
This section provides the cabling details for X9M and later models to connect four racks using RoCE Network Fabric.
Note:
-
The following conventions are used in the cabling notation for connecting multiple racks together:
-
The abbreviation for the first rack is R1, the second rack is R2, and so on.
-
LL identifies a lower leaf switch and UL identifies an upper leaf switch.
-
SS identifies the spine switch, which is located in U1 on all racks.
-
A specific switch is identified by combining abbreviations. For example, R1LL identifies the lower leaf switch (LL) on the first rack (R1).
-
-
The leaf switches are located as follows:
-
At rack unit 20 (U20) and 22 (U22) in 2-socket systems (Oracle Exadata Rack X9M-2 and later models).
-
At rack unit 21 (U21) and rack unit 23 (U23) in 8-socket systems (Oracle Exadata X9M-8).
-
-
The cable lengths shown in the following lists assume that the racks are adjacent to each other, the cables are routed through a raised floor, and there are no obstacles in the routing between the racks. If the racks are not adjacent, or use overhead cabling trays, then they may require longer cable lengths. Cable lengths up to 100 meters are supported.
-
Only optical cables (with additional transceivers) are supported for lengths greater than 5 meters.
-
For X9M-8 systems with three database servers and 11 storage servers only, port 30 on the leaf switches is connected to a database server and is not used as an inter-switch link. Consequently, for these systems only, ignore the connections to port number 30 on every leaf switch in the following tables. This adjustment leaves only 13 inter-switch links on each leaf switch and only applies to X9M-8 systems with three database servers and 11 storage servers.
The following tables contain details for all of the RoCE Network Fabric cabling connections in a four-rack system.
Table 4-8 Leaf Switch Connections for the First Rack in a Four-Rack System
| Switch Pair | Connections | Cable Length |
|---|---|---|
|
R1UL to R1SS |
R1UL-P2 to R1SS-P21 R1UL-P3 to R1SS-P23 R1UL-P5 to R1SS-P5 R1UL-P7 to R1SS-P7 |
3 meters |
|
R1UL to R2SS |
R1UL-P31 to R2SS-P5 R1UL-P33 to R2SS-P7 R1UL-P34 to R2SS-P21 R1UL-P35 to R2SS-P23 |
5 meters |
|
R1UL to R3SS |
R1UL-P1 to R3SS-P21 R1UL-P6 to R3SS-P5 R1UL-P32 to R3SS-P7 |
5 meters |
|
R1UL to R4SS |
R1UL-P4 to R4SS-P5 R1UL-P30 to R4SS-P7 R1UL-P36 to R4SS-P21 |
10 meters |
|
R1LL to R1SS |
R1LL-P3 to R1SS-P27 R1LL-P5 to R1SS-P13 R1LL-P7 to R1SS-P15 |
3 meters |
|
R1LL to R2SS |
R1LL-P31 to R2SS-P13 R1LL-P33 to R2SS-P15 R1LL-P34 to R2SS-P27 R1LL-P35 to R2SS-P29 |
5 meters |
|
R1LL to R3SS |
R1LL-P1 to R3SS-P27 R1LL-P2 to R3SS-P29 R1LL-P4 to R3SS-P13 R1LL-P6 to R3SS-P15 |
5 meters |
|
R1LL to R4SS |
R1LL-P30 to R4SS-P13 R1LL-P32 to R4SS-P15 R1LL-P36 to R4SS-P27 |
10 meters |
Table 4-9 Leaf Switch Connections for the Second Rack in a Four-Rack System
| Switch Pair | Connections | Cable Length |
|---|---|---|
|
R2UL to R1SS |
R2UL-P3 to R1SS-P22 R2UL-P5 to R1SS-P6 R2UL-P7 to R1SS-P8 |
5 meters |
|
R2UL to R2SS |
R2UL-P31 to R2SS-P6 R2UL-P33 to R2SS-P8 R2UL-P35 to R2SS-P22 |
3 meters |
|
R2UL to R3SS |
R2UL-P1 to R3SS-P22 R2UL-P2 to R3SS-P24 R2UL-P6 to R3SS-P6 R2UL-P30 to R3SS-P8 |
5 meters |
|
R2UL to R4SS |
R2UL-P4 to R4SS-P6 R2UL-P32 to R4SS-P8 R2UL-P34 to R4SS-P22 R2UL-P36 to R4SS-P24 |
5 meters |
|
R2LL to R1SS |
R2LL-P1 to R1SS-P24 R2LL-P3 to R1SS-P28 R2LL-P5 to R1SS-P14 R2LL-P7 to R1SS-P16 |
5 meters |
|
R2LL to R2SS |
R2LL-P31 to R2SS-P14 R2LL-P33 to R2SS-P16 R2LL-P35 to R2SS-P28 |
3 meters |
|
R2LL to R3SS |
R2LL-P6 to R3SS-P14 R2LL-P32 to R3SS-P16 R2LL-P34 to R3SS-P28 |
5 meters |
|
R2LL to R4SS |
R2LL-P2 to R4SS-P28 R2LL-P4 to R4SS-P14 R2LL-P30 to R4SS-P16 R2LL-P36 to R4SS-P30 |
5 meters |
Table 4-10 Leaf Switch Connections for the Third Rack in a Four-Rack System
| Switch Pair | Connections | Cable Length |
|---|---|---|
|
R3UL to R1SS |
R3UL-P1 to R1SS-P25 R3UL-P3 to R1SS-P29 R3UL-P5 to R1SS-P11 R3UL-P7 to R1SS-P12 |
5 meters |
|
R3UL to R2SS |
R3UL-P2 to R2SS-P25 R3UL-P6 to R2SS-P11 R3UL-P31 to R2SS-P12 R3UL-P35 to R2SS-P31 |
5 meters |
|
R3UL to R3SS |
R3UL-P30 to R3SS-P11 R3UL-P32 to R3SS-P12 R3UL-P34 to R3SS-P25 |
3 meters |
|
R3UL to R4SS |
R3UL-P4 to R4SS-P11 R3UL-P33 to R4SS-P12 R3UL-P36 to R4SS-P25 |
5 meters |
|
R3LL to R1SS |
R3LL-P3 to R1SS-P26 R3LL-P4 to R1SS-P20 R3LL-P7 to R1SS-P19 |
5 meters |
|
R3LL to R2SS |
R3LL-P2 to R2SS-P24 R3LL-P6 to R2SS-P19 R3LL-P31 to R2SS-P20 R3LL-P35 to R2SS-P26 |
5 meters |
|
R3LL to R3SS |
R3LL-P1 to R3SS-P26 R3LL-P30 to R3SS-P19 R3LL-P32 to R3SS-P20 R3LL-P34 to R3SS-P23 |
3 meters |
|
R3LL to R4SS |
R3LL-P5 to R4SS-P19 R3LL-P33 to R4SS-P20 R3LL-P36 to R4SS-P26 |
5 meters |
Table 4-11 Leaf Switch Connections for the Fourth Rack in a Four-Rack System
| Switch Pair | Connections | Cable Length |
|---|---|---|
|
R4UL to R1SS |
R4UL-P3 to R1SS-P31 R4UL-P5 to R1SS-P9 R4UL-P7 to R1SS-P10 |
10 meters |
|
R4UL to R2SS |
R4UL-P2 to R2SS-P30 R4UL-P4 to R2SS-P9 R4UL-P6 to R2SS-P10 |
5 meters |
|
R4UL to R3SS |
R4UL-P31 to R3SS-P9 R4UL-P33 to R3SS-P10 R4UL-P34 to R3SS-P30 R4UL-P35 to R3SS-P31 |
5 meters |
|
R4UL to R4SS |
R4UL-P1 to R4SS-P23 R4UL-P30 to R4SS-P9 R4UL-P32 to R4SS-P10 R4UL-P36 to R4SS-P29 |
3 meters |
|
R4LL to R1SS |
R4LL-P1 to R1SS-P32 R4LL-P3 to R1SS-P30 R4LL-P5 to R1SS-P17 R4LL-P7 to R1SS-P18 |
10 meters |
|
R4LL to R2SS |
R4LL-P2 to R2SS-P32 R4LL-P4 to R2SS-P17 R4LL-P6 to R2SS-P18 |
5 meters |
|
R4LL to R3SS |
R4LL-P31 to R3SS-P17 R4LL-P33 to R3SS-P18 R4LL-P34 to R3SS-P32 |
5 meters |
|
R4LL to R4SS |
R4LL-P30 to R4SS-P17 R4LL-P32 to R4SS-P18 R4LL-P35 to R4SS-P31 R4LL-P36 to R4SS-P32 |
3 meters |
The following table contains all of the RoCE Network Fabric cabling connections from the previous tables. In this table, the connections are sorted by the spine switch port location.
Table 4-12 Four-Rack System Connections Sorted By The Spine Switch Port Location
| Spine Switch Port | Leaf Switch Port | Cable Length |
|---|---|---|
| R1SS-P5 | R1UL-P5 | 3 meters |
| R1SS-P6 | R2UL-P5 | 5 meters |
| R1SS-P7 | R1UL-P7 | 3 meters |
| R1SS-P8 | R2UL-P7 | 5 meters |
| R1SS-P9 | R4UL-P5 | 10 meters |
| R1SS-P10 | R4UL-P7 | 10 meters |
| R1SS-P11 | R3UL-P5 | 5 meters |
| R1SS-P12 | R3UL-P7 | 5 meters |
| R1SS-P13 | R1LL-P5 | 3 meters |
| R1SS-P14 | R2LL-P5 | 5 meters |
| R1SS-P15 | R1LL-P7 | 3 meters |
| R1SS-P16 | R2LL-P7 | 5 meters |
| R1SS-P17 | R4LL-P5 | 10 meters |
| R1SS-P18 | R4LL-P7 | 10 meters |
| R1SS-P19 | R3LL-P7 | 5 meters |
| R1SS-P20 | R3LL-P4 | 5 meters |
| R1SS-P21 | R1UL-P2 | 3 meters |
| R1SS-P22 | R2UL-P3 | 5 meters |
| R1SS-P23 | R1UL-P3 | 3 meters |
| R1SS-P24 | R2LL-P1 | 5 meters |
| R1SS-P25 | R3UL-P1 | 5 meters |
| R1SS-P26 | R3LL-P3 | 5 meters |
| R1SS-P27 | R1LL-P3 | 3 meters |
| R1SS-P28 | R2LL-P3 | 5 meters |
| R1SS-P29 | R3UL-P3 | 5 meters |
| R1SS-P30 | R4LL-P3 | 10 meters |
| R1SS-P31 | R4UL-P3 | 10 meters |
| R1SS-P32 | R4LL-P1 | 10 meters |
| R2SS-P5 | R1UL-P31 | 5 meters |
| R2SS-P6 | R2UL-P31 | 3 meters |
| R2SS-P7 | R1UL-P33 | 5 meters |
| R2SS-P8 | R2UL-P33 | 3 meters |
| R2SS-P9 | R4UL-P4 | 5 meters |
| R2SS-P10 | R4UL-P6 | 5 meters |
| R2SS-P11 | R3UL-P6 | 5 meters |
| R2SS-P12 | R3UL-P31 | 5 meters |
| R2SS-P13 | R1LL-P31 | 5 meters |
| R2SS-P14 | R2LL-P31 | 3 meters |
| R2SS-P15 | R1LL-P33 | 5 meters |
| R2SS-P16 | R2LL-P33 | 3 meters |
| R2SS-P17 | R4LL-P4 | 5 meters |
| R2SS-P18 | R4LL-P6 | 5 meters |
| R2SS-P19 | R3LL-P6 | 5 meters |
| R2SS-P20 | R3LL-P31 | 5 meters |
| R2SS-P21 | R1UL-P34 | 5 meters |
| R2SS-P22 | R2UL-P35 | 3 meters |
| R2SS-P23 | R1UL-P35 | 5 meters |
| R2SS-P24 | R3LL-P2 | 5 meters |
| R2SS-P25 | R3UL-P2 | 5 meters |
| R2SS-P26 | R3LL-P35 | 5 meters |
| R2SS-P27 | R1LL-P34 | 5 meters |
| R2SS-P28 | R2LL-P35 | 3 meters |
| R2SS-P29 | R1LL-P35 | 5 meters |
| R2SS-P30 | R4UL-P2 | 5 meters |
| R2SS-P31 | R3UL-P35 | 5 meters |
| R2SS-P32 | R4LL-P2 | 5 meters |
| R3SS-P5 | R1UL-P6 | 5 meters |
| R3SS-P6 | R2UL-P6 | 5 meters |
| R3SS-P7 | R1UL-P32 | 5 meters |
| R3SS-P8 | R2UL-P30 | 5 meters |
| R3SS-P9 | R4UL-P31 | 5 meters |
| R3SS-P10 | R4UL-P33 | 5 meters |
| R3SS-P11 | R3UL-P30 | 3 meters |
| R3SS-P12 | R3UL-P32 | 3 meters |
| R3SS-P13 | R1LL-P4 | 5 meters |
| R3SS-P14 | R2LL-P6 | 5 meters |
| R3SS-P15 | R1LL-P6 | 5 meters |
| R3SS-P16 | R2LL-P32 | 5 meters |
| R3SS-P17 | R4LL-P31 | 5 meters |
| R3SS-P18 | R4LL-P33 | 5 meters |
| R3SS-P19 | R3LL-P30 | 3 meters |
| R3SS-P20 | R3LL-P32 | 3 meters |
| R3SS-P21 | R1UL-P1 | 5 meters |
| R3SS-P22 | R2UL-P1 | 5 meters |
| R3SS-P23 | R3LL-P34 | 3 meters |
| R3SS-P24 | R2UL-P2 | 5 meters |
| R3SS-P25 | R3UL-P34 | 3 meters |
| R3SS-P26 | R3LL-P1 | 3 meters |
| R3SS-P27 | R1LL-P1 | 5 meters |
| R3SS-P28 | R2LL-P34 | 5 meters |
| R3SS-P29 | R1LL-P2 | 5 meters |
| R3SS-P30 | R4UL-P34 | 5 meters |
| R3SS-P31 | R4UL-P35 | 5 meters |
| R3SS-P32 | R4LL-P34 | 5 meters |
| R4SS-P5 | R1UL-P4 | 10 meters |
| R4SS-P6 | R2UL-P4 | 5 meters |
| R4SS-P7 | R1UL-P30 | 10 meters |
| R4SS-P8 | R2UL-P32 | 5 meters |
| R4SS-P9 | R4UL-P30 | 3 meters |
| R4SS-P10 | R4UL-P32 | 3 meters |
| R4SS-P11 | R3UL-P4 | 5 meters |
| R4SS-P12 | R3UL-P33 | 5 meters |
| R4SS-P13 | R1LL-P30 | 10 meters |
| R4SS-P14 | R2LL-P4 | 5 meters |
| R4SS-P15 | R1LL-P32 | 10 meters |
| R4SS-P16 | R2LL-P30 | 5 meters |
| R4SS-P17 | R4LL-P30 | 3 meters |
| R4SS-P18 | R4LL-P32 | 3 meters |
| R4SS-P19 | R3LL-P5 | 5 meters |
| R4SS-P20 | R3LL-P33 | 5 meters |
| R4SS-P21 | R1UL-P36 | 10 meters |
| R4SS-P22 | R2UL-P34 | 5 meters |
| R4SS-P23 | R4UL-P1 | 3 meters |
| R4SS-P24 | R2UL-P36 | 5 meters |
| R4SS-P25 | R3UL-P36 | 5 meters |
| R4SS-P26 | R3LL-P36 | 5 meters |
| R4SS-P27 | R1LL-P36 | 10 meters |
| R4SS-P28 | R2LL-P2 | 5 meters |
| R4SS-P29 | R4UL-P36 | 3 meters |
| R4SS-P30 | R2LL-P36 | 5 meters |
| R4SS-P31 | R4LL-P35 | 3 meters |
| R4SS-P32 | R4LL-P36 | 3 meters |