4.7 Six-Rack Cabling for X9M and Later Model Racks
This section provides the cabling details for X9M and later models to connect six racks using RoCE Network Fabric.
Note:
-
The following conventions are used in the cabling notation for connecting multiple racks together:
-
The abbreviation for the first rack is R1, the second rack is R2, and so on.
-
LL identifies a lower leaf switch and UL identifies an upper leaf switch.
-
SS identifies the spine switch, which is located in U1 on all racks.
-
A specific switch is identified by combining abbreviations. For example, R1LL identifies the lower leaf switch (LL) on the first rack (R1).
-
-
The leaf switches are located as follows:
-
At rack unit 20 (U20) and 22 (U22) in 2-socket systems (Oracle Exadata Rack X9M-2 and later models).
-
At rack unit 21 (U21) and rack unit 23 (U23) in 8-socket systems (Oracle Exadata X9M-8).
-
-
The cable lengths shown in the following lists assume that the racks are adjacent to each other, the cables are routed through a raised floor, and there are no obstacles in the routing between the racks. If the racks are not adjacent, or use overhead cabling trays, then they may require longer cable lengths. Cable lengths up to 100 meters are supported.
-
Only optical cables (with additional transceivers) are supported for lengths greater than 5 meters.
-
For X9M-8 systems with three database servers and 11 storage servers only, port 30 on the leaf switches is connected to a database server and is not used as an inter-switch link. Consequently, for these systems only, ignore the connections to port number 30 on every leaf switch in the following tables. This adjustment leaves only 13 inter-switch links on each leaf switch and only applies to X9M-8 systems with three database servers and 11 storage servers.
The following tables contain details for all of the RoCE Network Fabric cabling connections in a six-rack system.
Table 4-19 Leaf Switch Connections for the First Rack in a Six-Rack System
Switch Pair | Connections | Cable Length |
---|---|---|
R1UL to R1SS |
R1UL-P3 to R1SS-P21 R1UL-P5 to R1SS-P5 R1UL-P7 to R1SS-P7 |
3 meters |
R1UL to R2SS |
R1UL-P31 to R2SS-P5 R1UL-P33 to R2SS-P7 R1UL-P35 to R2SS-P21 |
5 meters |
R1UL to R3SS |
R1UL-P1 to R3SS-P21 R1UL-P6 to R3SS-P5 |
5 meters |
R1UL to R4SS |
R1UL-P2 to R4SS-P21 R1UL-P4 to R4SS-P5 |
10 meters |
R1UL to R5SS |
R1UL-P30 to R5SS-P5 R1UL-P34 to R5SS-P21 |
10 meters |
R1UL to R6SS |
R1UL-P32 to R6SS-P5 R1UL-P36 to R6SS-P21 |
10 meters |
R1LL to R1SS |
R1LL-P3 to R1SS-P27 R1LL-P5 to R1SS-P13 |
3 meters |
R1LL to R2SS |
R1LL-P31 to R2SS-P13 R1LL-P33 to R2SS-P15 R1LL-P35 to R2SS-P27 |
5 meters |
R1LL to R3SS |
R1LL-P2 to R3SS-P27 R1LL-P4 to R3SS-P13 R1LL-P6 to R3SS-P15 |
5 meters |
R1LL to R4SS |
R1LL-P30 to R4SS-P13 R1LL-P34 to R4SS-P27 |
10 meters |
R1LL to R5SS |
R1LL-P7 to R5SS-P13 R1LL-P1 to R5SS-P27 |
10 meters |
R1LL to R6SS |
R1LL-P32 to R6SS-P13 R1LL-P36 to R6SS-P27 |
10 meters |
Table 4-20 Leaf Switch Connections for the Second Rack in a Six-Rack System
Switch Pair | Connections | Cable Length |
---|---|---|
R2UL to R1SS |
R2UL-P3 to R1SS-P22 R2UL-P5 to R1SS-P6 |
5 meters |
R2UL to R2SS |
R2UL-P31 to R2SS-P6 R2UL-P35 to R2SS-P22 |
3 meters |
R2UL to R3SS |
R2UL-P6 to R3SS-P6 R2UL-P30 to R3SS-P8 R2UL-P2 to R3SS-P22 |
5 meters |
R2UL to R4SS |
R2UL-P4 to R4SS-P6 R2UL-P32 to R4SS-P8 R2UL-P34 to R4SS-P22 |
5 meters |
R2UL to R5SS |
R2UL-P1 to R5SS-P22 R2UL-P7 to R5SS-P6 |
10 meters |
R2UL to R6SS |
R2UL-P33 to R6SS-P6 R2UL-P36 to R6SS-P22 |
10 meters |
R2LL to R1SS |
R2LL-P3 to R1SS-P28 R2LL-P5 to R1SS-P14 |
5 meters |
R2LL to R2SS |
R2LL-P31 to R2SS-P14 R2LL-P35 to R2SS-P28 |
3 meters |
R2LL to R3SS |
R2LL-P2 to R3SS-P28 R2LL-P6 to R3SS-P14 |
5 meters |
R2LL to R4SS |
R2LL-P4 to R4SS-P14 R2LL-P30 to R4SS-P16 R2LL-P34 to R4SS-P28 |
5 meters |
R2LL to R5SS |
R2LL-P32 to R5SS-P16 R2LL-P33 to R5SS-P14 R2LL-P36 to R5SS-P28 |
10 meters |
R2LL to R6SS |
R2LL-P1 to R6SS-P28 R2LL-P7 to R6SS-P14 |
10 meters |
Table 4-21 Leaf Switch Connections for the Third Rack in a Six-Rack System
Switch Pair | Connections | Cable Length |
---|---|---|
R3UL to R1SS |
R3UL-P3 to R1SS-P25 R3UL-P5 to R1SS-P11 |
5 meters |
R3UL to R2SS |
R3UL-P2 to R2SS-P25 R3UL-P6 to R2SS-P11 |
5 meters |
R3UL to R3SS |
R3UL-P30 to R3SS-P11 R3UL-P34 to R3SS-P25 |
3 meters |
R3UL to R4SS |
R3UL-P1 to R4SS-P25 R3UL-P4 to R4SS-P11 |
5 meters |
R3UL to R5SS |
R3UL-P32 to R5SS-P11 R3UL-P33 to R5SS-P12 R3UL-P35 to R5SS-P25 |
5 meters |
R3UL to R6SS |
R3UL-P7 to R6SS-P11 R3UL-P31 to R6SS-P12 R3UL-P36 to R6SS-P25 |
10 meters |
R3LL to R1SS |
R3LL-P3 to R1SS-P26 R3LL-P4 to R1SS-P20 R3LL-P7 to R1SS-P19 |
5 meters |
R3LL to R2SS |
R3LL-P31 to R2SS-P20 R3LL-P35 to R2SS-P26 |
5 meters |
R3LL to R3SS |
R3LL-P32 to R3SS-P20 R3LL-P34 to R3SS-P26 |
3 meters |
R3LL to R4SS |
R3LL-P33 to R4SS-P20 R3LL-P36 to R4SS-P26 |
5 meters |
R3LL to R5SS |
R3LL-P1 to R5SS-P26 R3LL-P5 to R5SS-P20 |
5 meters |
R3LL to R6SS |
R3LL-P2 to R6SS-P26 R3LL-P6 to R6SS-P19 R3LL-P30 to R6SS-P20 |
10 meters |
Table 4-22 Leaf Switch Connections for the Fourth Rack in a Six-Rack System
Switch Pair | Connections | Cable Length |
---|---|---|
R4UL to R1SS |
R4UL-P1 to R1SS-P32 R4UL-P3 to R1SS-P31 R4UL-P7 to R1SS-P10 |
10 meters |
R4UL to R2SS |
R4UL-P2 to R2SS-P31 R4UL-P4 to R2SS-P9 R4UL-P6 to R2SS-P10 |
5 meters |
R4UL to R3SS |
R4UL-P31 to R3SS-P9 R4UL-P33 to R3SS-P10 |
5 meters |
R4UL to R4SS |
R4UL-P32 to R4SS-P10 R4UL-P34 to R4SS-P31 |
3 meters |
R4UL to R5SS |
R4UL-P5 to R5SS-P10 R4UL-P35 to R5SS-P31 |
5 meters |
R4UL to R6SS |
R4UL-P30 to R6SS-P10 R4UL-P36 to R6SS-P31 |
5 meters |
R4LL to R1SS |
R4LL-P3 to R1SS-P30 R4LL-P5 to R1SS-P17 |
10 meters |
R4LL to R2SS |
R4LL-P1 to R2SS-P32 R4LL-P2 to R2SS-P30 R4LL-P4 to R2SS-P17 |
5 meters |
R4LL to R3SS |
R4LL-P31 to R3SS-P17 R4LL-P33 to R3SS-P18 R4LL-P35 to R3SS-P30 |
5 meters |
R4LL to R4SS |
R4LL-P30 to R4SS-P17 R4LL-P32 to R4SS-P18 |
3 meters |
R4LL to R5SS |
R4LL-P7 to R5SS-P17 R4LL-P34 to R5SS-P30 |
5 meters |
R4LL to R6SS |
R4LL-P6 to R6SS-P17 R4LL-P36 to R6SS-P30 |
5 meters |
Table 4-23 Leaf Switch Connections for the Fifth Rack in a Six-Rack System
Switch Pair | Connections | Cable Length |
---|---|---|
R5UL to R1SS |
R5UL-P3 to R1SS-P23 R5UL-P5 to R1SS-P8 |
10 meters |
R5UL to R2SS |
R5UL-P2 to R2SS-P23 R5UL-P4 to R2SS-P8 |
10 meters |
R5UL to R3SS |
R5UL-P6 to R3SS-P12 R5UL-P1 to R3SS-P23 R5UL-P34 to R3SS-P24 |
5 meters |
R5UL to R4SS |
R5UL-P31 to R4SS-P7 R5UL-P33 to R4SS-P12 R5UL-P35 to R4SS-P23 |
5 meters |
R5UL to R5SS |
R5UL-P30 to R5SS-P7 R5UL-P32 to R5SS-P9 |
3 meters |
R5UL to R6SS |
R5UL-P7 to R6SS-P7 R5UL-P36 to R6SS-P23 |
5 meters |
R5LL to R1SS |
R5LL-P3 to R1SS-P29 R5LL-P7 to R1SS-P18 |
10 meters |
R5LL to R2SS |
R5LL-P6 to R2SS-P18 R5LL-P34 to R2SS-P29 |
10 meters |
R5LL to R3SS |
R5LL-P31 to R3SS-P16 R5LL-P35 to R3SS-P29 |
5 meters |
R5LL to R4SS |
R5LL-P1 to R4SS-P29 R5LL-P2 to R4SS-P30 R5LL-P33 to R4SS-P19 |
5 meters |
R5LL to R5SS |
R5LL-P30 to R5SS-P15 R5LL-P32 to R5SS-P19 R5LL-P36 to R5SS-P29 |
3 meters |
R5LL to R6SS |
R5LL-P4 to R6SS-P15 R5LL-P5 to R6SS-P9 |
5 meters |
Table 4-24 Leaf Switch Connections for the Sixth Rack in a Six-Rack System
Switch Pair | Connections | Cable Length |
---|---|---|
R6UL to R1SS |
R6UL-P5 to R1SS-P9 R6UL-P7 to R1SS-P16 |
10 meters |
R6UL to R2SS |
R6UL-P2 to R2SS-P24 R6UL-P4 to R2SS-P16 |
10 meters |
R6UL to R3SS |
R6UL-P3 to R3SS-P31 R6UL-P6 to R3SS-P19 |
10 meters |
R6UL to R4SS |
R6UL-P1 to R4SS-P32 R6UL-P31 to R4SS-P15 |
5 meters |
R6UL to R5SS |
R6UL-P33 to R5SS-P18 R6UL-P34 to R5SS-P24 R6UL-P35 to R5SS-P23 |
5 meters |
R6UL to R6SS |
R6UL-P30 to R6SS-P16 R6UL-P32 to R6SS-P18 R6UL-P36 to R6SS-P24 |
3 meters |
R6LL to R1SS |
R6LL-P3 to R1SS-P24 R6LL-P5 to R1SS-P12 R6LL-P7 to R1SS-P15 |
10 meters |
R6LL to R2SS |
R6LL-P4 to R2SS-P12 R6LL-P6 to R2SS-P19 |
10 meters |
R6LL to R3SS |
R6LL-P1 to R3SS-P32 R6LL-P31 to R3SS-P7 |
10 meters |
R6LL to R4SS |
R6LL-P2 to R4SS-P24 R6LL-P33 to R4SS-P9 |
5 meters |
R6LL to R5SS |
R6LL-P30 to R5SS-P8 R6LL-P34 to R5SS-P32 |
5 meters |
R6LL to R6SS |
R6LL-P32 to R6SS-P8 R6LL-P35 to R6SS-P29 R6LL-P36 to R6SS-P32 |
3 meters |
The following table contains all of the RoCE Network Fabric cabling connections from the previous tables. In this table, the connections are sorted by the spine switch port location.
Table 4-25 Six-Rack System Connections Sorted By The Spine Switch Port Location
Spine Switch Port | Leaf Switch Port | Cable Length |
---|---|---|
R1SS-P5 | R1UL-P5 | 3 meters |
R1SS-P6 | R2UL-P5 | 5 meters |
R1SS-P7 | R1UL-P7 | 3 meters |
R1SS-P8 | R5UL-P5 | 10 meters |
R1SS-P9 | R6UL-P5 | 10 meters |
R1SS-P10 | R4UL-P7 | 10 meters |
R1SS-P11 | R3UL-P5 | 5 meters |
R1SS-P12 | R6LL-P5 | 10 meters |
R1SS-P13 | R1LL-P5 | 3 meters |
R1SS-P14 | R2LL-P5 | 5 meters |
R1SS-P15 | R6LL-P7 | 10 meters |
R1SS-P16 | R6UL-P7 | 10 meters |
R1SS-P17 | R4LL-P5 | 10 meters |
R1SS-P18 | R5LL-P7 | 10 meters |
R1SS-P19 | R3LL-P7 | 5 meters |
R1SS-P20 | R3LL-P4 | 5 meters |
R1SS-P21 | R1UL-P3 | 3 meters |
R1SS-P22 | R2UL-P3 | 5 meters |
R1SS-P23 | R5UL-P3 | 10 meters |
R1SS-P24 | R6LL-P3 | 10 meters |
R1SS-P25 | R3UL-P3 | 5 meters |
R1SS-P26 | R3LL-P3 | 5 meters |
R1SS-P27 | R1LL-P3 | 3 meters |
R1SS-P28 | R2LL-P3 | 5 meters |
R1SS-P29 | R5LL-P3 | 10 meters |
R1SS-P30 | R4LL-P3 | 10 meters |
R1SS-P31 | R4UL-P3 | 10 meters |
R1SS-P32 | R4UL-P1 | 10 meters |
R2SS-P5 | R1UL-P31 | 5 meters |
R2SS-P6 | R2UL-P31 | 3 meters |
R2SS-P7 | R1UL-P33 | 5 meters |
R2SS-P8 | R5UL-P4 | 10 meters |
R2SS-P9 | R4UL-P4 | 5 meters |
R2SS-P10 | R4UL-P6 | 5 meters |
R2SS-P11 | R3UL-P6 | 5 meters |
R2SS-P12 | R6LL-P4 | 10 meters |
R2SS-P13 | R1LL-P31 | 5 meters |
R2SS-P14 | R2LL-P31 | 3 meters |
R2SS-P15 | R1LL-P33 | 5 meters |
R2SS-P16 | R6UL-P4 | 10 meters |
R2SS-P17 | R4LL-P4 | 5 meters |
R2SS-P18 | R5LL-P6 | 10 meters |
R2SS-P19 | R6LL-P6 | 10 meters |
R2SS-P20 | R3LL-P31 | 5 meters |
R2SS-P21 | R1UL-P35 | 5 meters |
R2SS-P22 | R2UL-P35 | 3 meters |
R2SS-P23 | R5UL-P2 | 10 meters |
R2SS-P24 | R6UL-P2 | 10 meters |
R2SS-P25 | R3UL-P2 | 5 meters |
R2SS-P26 | R3LL-P35 | 5 meters |
R2SS-P27 | R1LL-P35 | 5 meters |
R2SS-P28 | R2LL-P35 | 3 meters |
R2SS-P29 | R5LL-P34 | 10 meters |
R2SS-P30 | R4LL-P2 | 5 meters |
R2SS-P31 | R4UL-P2 | 5 meters |
R2SS-P32 | R4LL-P1 | 5 meters |
R3SS-P5 | R1UL-P6 | 5 meters |
R3SS-P6 | R2UL-P6 | 5 meters |
R3SS-P7 | R6LL-P31 | 10 meters |
R3SS-P8 | R2UL-P30 | 5 meters |
R3SS-P9 | R4UL-P31 | 5 meters |
R3SS-P10 | R4UL-P33 | 5 meters |
R3SS-P11 | R3UL-P30 | 3 meters |
R3SS-P12 | R5UL-P6 | 5 meters |
R3SS-P13 | R1LL-P4 | 5 meters |
R3SS-P14 | R2LL-P6 | 5 meters |
R3SS-P15 | R1LL-P6 | 5 meters |
R3SS-P16 | R5LL-P31 | 5 meters |
R3SS-P17 | R4LL-P31 | 5 meters |
R3SS-P18 | R4LL-P33 | 5 meters |
R3SS-P19 | R6UL-P6 | 10 meters |
R3SS-P20 | R3LL-P32 | 3 meters |
R3SS-P21 | R1UL-P1 | 5 meters |
R3SS-P22 | R2UL-P2 | 5 meters |
R3SS-P23 | R5UL-P1 | 5 meters |
R3SS-P24 | R5UL-P34 | 5 meters |
R3SS-P25 | R3UL-P34 | 3 meters |
R3SS-P26 | R3LL-P34 | 3 meters |
R3SS-P27 | R1LL-P2 | 5 meters |
R3SS-P28 | R2LL-P2 | 5 meters |
R3SS-P29 | R5LL-P35 | 5 meters |
R3SS-P30 | R4LL-P35 | 5 meters |
R3SS-P31 | R6UL-P3 | 10 meters |
R3SS-P32 | R6LL-P1 | 10 meters |
R4SS-P5 | R1UL-P4 | 10 meters |
R4SS-P6 | R2UL-P4 | 5 meters |
R4SS-P7 | R5UL-P31 | 5 meters |
R4SS-P8 | R2UL-P32 | 5 meters |
R4SS-P9 | R6LL-P33 | 5 meters |
R4SS-P10 | R4UL-P32 | 3 meters |
R4SS-P11 | R3UL-P4 | 5 meters |
R4SS-P12 | R5UL-P33 | 5 meters |
R4SS-P13 | R1LL-P30 | 10 meters |
R4SS-P14 | R2LL-P4 | 5 meters |
R4SS-P15 | R6UL-P31 | 5 meters |
R4SS-P16 | R2LL-P30 | 5 meters |
R4SS-P17 | R4LL-P30 | 3 meters |
R4SS-P18 | R4LL-P32 | 3 meters |
R4SS-P19 | R5LL-P33 | 5 meters |
R4SS-P20 | R3LL-P33 | 5 meters |
R4SS-P21 | R1UL-P2 | 10 meters |
R4SS-P22 | R2UL-P34 | 5 meters |
R4SS-P23 | R5UL-P35 | 5 meters |
R4SS-P24 | R6LL-P2 | 5 meters |
R4SS-P25 | R3UL-P1 | 5 meters |
R4SS-P26 | R3LL-P36 | 5 meters |
R4SS-P27 | R1LL-P34 | 10 meters |
R4SS-P28 | R2LL-P34 | 5 meters |
R4SS-P29 | R5LL-P1 | 5 meters |
R4SS-P30 | R5LL-P2 | 5 meters |
R4SS-P31 | R4UL-P34 | 3 meters |
R4SS-P32 | R6UL-P1 | 5 meters |
R5SS-P5 | R1UL-P30 | 10 meters |
R5SS-P6 | R2UL-P7 | 10 meters |
R5SS-P7 | R5UL-P30 | 3 meters |
R5SS-P8 | R6LL-P30 | 5 meters |
R5SS-P9 | R5UL-P32 | 3 meters |
R5SS-P10 | R4UL-P5 | 5 meters |
R5SS-P11 | R3UL-P32 | 5 meters |
R5SS-P12 | R3UL-P33 | 5 meters |
R5SS-P13 | R1LL-P7 | 10 meters |
R5SS-P14 | R2LL-P33 | 10 meters |
R5SS-P15 | R5LL-P30 | 3 meters |
R5SS-P16 | R2LL-P32 | 10 meters |
R5SS-P17 | R4LL-P7 | 5 meters |
R5SS-P18 | R6UL-P33 | 5 meters |
R5SS-P19 | R5LL-P32 | 3 meters |
R5SS-P20 | R3LL-P5 | 5 meters |
R5SS-P21 | R1UL-P34 | 10 meters |
R5SS-P22 | R2UL-P1 | 10 meters |
R5SS-P23 | R6UL-P35 | 5 meters |
R5SS-P24 | R6UL-P34 | 5 meters |
R5SS-P25 | R3UL-P35 | 5 meters |
R5SS-P26 | R3LL-P1 | 5 meters |
R5SS-P27 | R1LL-P1 | 10 meters |
R5SS-P28 | R2LL-P36 | 10 meters |
R5SS-P29 | R5LL-P36 | 3 meters |
R5SS-P30 | R4LL-P34 | 5 meters |
R5SS-P31 | R4UL-P35 | 5 meters |
R5SS-P32 | R6LL-P34 | 5 meters |
R6SS-P5 | R1UL-P32 | 10 meters |
R6SS-P6 | R2UL-P33 | 10 meters |
R6SS-P7 | R5UL-P7 | 5 meters |
R6SS-P8 | R6LL-P32 | 3 meters |
R6SS-P9 | R5LL-P5 | 5 meters |
R6SS-P10 | R4UL-P30 | 5 meters |
R6SS-P11 | R3UL-P7 | 10 meters |
R6SS-P12 | R3UL-P31 | 10 meters |
R6SS-P13 | R1LL-P32 | 10 meters |
R6SS-P14 | R2LL-P7 | 10 meters |
R6SS-P15 | R5LL-P4 | 5 meters |
R6SS-P16 | R6UL-P30 | 3 meters |
R6SS-P17 | R4LL-P6 | 5 meters |
R6SS-P18 | R6UL-P32 | 3 meters |
R6SS-P19 | R3LL-P6 | 10 meters |
R6SS-P20 | R3LL-P30 | 10 meters |
R6SS-P21 | R1UL-P36 | 10 meters |
R6SS-P22 | R2UL-P36 | 10 meters |
R6SS-P23 | R5UL-P36 | 5 meters |
R6SS-P24 | R6UL-P36 | 3 meters |
R6SS-P25 | R3UL-P36 | 10 meters |
R6SS-P26 | R3LL-P2 | 10 meters |
R6SS-P27 | R1LL-P36 | 10 meters |
R6SS-P28 | R2LL-P1 | 10 meters |
R6SS-P29 | R6LL-P35 | 3 meters |
R6SS-P30 | R4LL-P36 | 5 meters |
R6SS-P31 | R4UL-P36 | 5 meters |
R6SS-P32 | R6LL-P36 | 3 meters |