4.8 Seven-Rack Cabling for X9M and Later Model Racks

This section provides the cabling details for X9M and later models to connect seven racks using RoCE Network Fabric.

Note:

  • The following conventions are used in the cabling notation for connecting multiple racks together:

    • The abbreviation for the first rack is R1, the second rack is R2, and so on.

    • LL identifies a lower leaf switch and UL identifies an upper leaf switch.

    • SS identifies the spine switch, which is located in U1 on all racks.

    • A specific switch is identified by combining abbreviations. For example, R1LL identifies the lower leaf switch (LL) on the first rack (R1).

  • The leaf switches are located as follows:

    • At rack unit 20 (U20) and 22 (U22) in 2-socket systems (Oracle Exadata Rack X9M-2 and later models).

    • At rack unit 21 (U21) and rack unit 23 (U23) in 8-socket systems (Oracle Exadata X9M-8).

  • The cable lengths shown in the following lists assume that the racks are adjacent to each other, the cables are routed through a raised floor, and there are no obstacles in the routing between the racks. If the racks are not adjacent, or use overhead cabling trays, then they may require longer cable lengths. Cable lengths up to 100 meters are supported.

  • Only optical cables (with additional transceivers) are supported for lengths greater than 5 meters.

  • For X9M-8 systems with three database servers and 11 storage servers only, port 30 on the leaf switches is connected to a database server and is not used as an inter-switch link. Consequently, for these systems only, ignore the connections to port number 30 on every leaf switch in the following tables. This adjustment leaves only 13 inter-switch links on each leaf switch and only applies to X9M-8 systems with three database servers and 11 storage servers.

The following tables contain details for all of the RoCE Network Fabric cabling connections in a seven-rack system.

Table 4-26 Leaf Switch Connections for the First Rack in a Seven-Rack System

Switch Pair Connections Cable Length

R1UL to R1SS

R1UL-P5 to R1SS-P5

R1UL-P7 to R1SS-P7

3 meters

R1UL to R2SS

R1UL-P3 to R2SS-P21

R1UL-P31 to R2SS-P5

5 meters

R1UL to R3SS

R1UL-P1 to R3SS-P27

R1UL-P6 to R3SS-P5

5 meters

R1UL to R4SS

R1UL-P2 to R4SS-P22

R1UL-P4 to R4SS-P5

10 meters

R1UL to R5SS

R1UL-P30 to R5SS-P5

R1UL-P34 to R5SS-P29

10 meters

R1UL to R6SS

R1UL-P32 to R6SS-P5

R1UL-P35 to R6SS-P23

10 meters

R1UL to R7SS

R1UL-P33 to R7SS-P5

R1UL-P36 to R7SS-P23

10 meters

R1LL to R1SS

R1LL-P3 to R1SS-P21

R1LL-P5 to R1SS-P13

3 meters

R1LL to R2SS

R1LL-P31 to R2SS-P13

R1LL-P33 to R2SS-P15

5 meters

R1LL to R3SS

R1LL-P2 to R3SS-P21

R1LL-P4 to R3SS-P13

5 meters

R1LL to R4SS

R1LL-P30 to R4SS-P13

R1LL-P34 to R4SS-P21

10 meters

R1LL to R5SS

R1LL-P7 to R5SS-P13

R1LL-P35 to R5SS-P21

10 meters

R1LL to R6SS

R1LL-P32 to R6SS-P13

R1LL-P36 to R6SS-P21

10 meters

R1LL to R7SS

R1LL-P1 to R7SS-P21

R1LL-P6 to R7SS-P13

10 meters

Table 4-27 Leaf Switch Connections for the Second Rack in a Seven-Rack System

Switch Pair Connections Cable Length

R2UL to R1SS

R2UL-P3 to R1SS-P28

R2UL-P5 to R1SS-P6

5 meters

R2UL to R2SS

R2UL-P31 to R2SS-P6

R2UL-P34 to R2SS-P28

3 meters

R2UL to R3SS

R2UL-P6 to R3SS-P6

R2UL-P30 to R3SS-P8

5 meters

R2UL to R4SS

R2UL-P2 to R4SS-P28

R2UL-P4 to R4SS-P6

5 meters

R2UL to R5SS

R2UL-P1 to R5SS-P28

R2UL-P7 to R5SS-P6

10 meters

R2UL to R6SS

R2UL-P33 to R6SS-P6

R2UL-P35 to R6SS-P28

10 meters

R2UL to R7SS

R2UL-P32 to R7SS-P6

R2UL-P36 to R7SS-P28

10 meters

R2LL to R1SS

R2LL-P3 to R1SS-P22

R2LL-P5 to R1SS-P14

5 meters

R2LL to R2SS

R2LL-P2 to R2SS-P22

R2LL-P31 to R2SS-P14

3 meters

R2LL to R3SS

R2LL-P1 to R3SS-P22

R2LL-P6 to R3SS-P14

5 meters

R2LL to R4SS

R2LL-P4 to R4SS-P14

R2LL-P30 to R4SS-P16

5 meters

R2LL to R5SS

R2LL-P33 to R5SS-P14

R2LL-P35 to R5SS-P22

10 meters

R2LL to R6SS

R2LL-P7 to R6SS-P14

R2LL-P36 to R6SS-P22

10 meters

R2LL to R7SS

R2LL-P32 to R7SS-P14

R2LL-P34 to R7SS-P22

10 meters

Table 4-28 Leaf Switch Connections for the Third Rack in a Seven-Rack System

Switch Pair Connections Cable Length

R3UL to R1SS

R3UL-P3 to R1SS-P25

R3UL-P5 to R1SS-P11

5 meters

R3UL to R2SS

R3UL-P2 to R2SS-P25

R3UL-P6 to R2SS-P11

5 meters

R3UL to R3SS

R3UL-P30 to R3SS-P11

R3UL-P34 to R3SS-P25

3 meters

R3UL to R4SS

R3UL-P1 to R4SS-P25

R3UL-P4 to R4SS-P11

5 meters

R3UL to R5SS

R3UL-P32 to R5SS-P11

R3UL-P33 to R5SS-P12

5 meters

R3UL to R6SS

R3UL-P7 to R6SS-P11

R3UL-P36 to R6SS-P32

10 meters

R3UL to R7SS

R3UL-P31 to R7SS-P11

R3UL-P35 to R7SS-P12

10 meters

R3LL to R1SS

R3LL-P2 to R1SS-P32

R3LL-P4 to R1SS-P20

5 meters

R3LL to R2SS

R3LL-P1 to R2SS-P32

R3LL-P31 to R2SS-P20

5 meters

R3LL to R3SS

R3LL-P32 to R3SS-P20

R3LL-P34 to R3SS-P32

3 meters

R3LL to R4SS

R3LL-P33 to R4SS-P20

R3LL-P35 to R4SS-P32

5 meters

R3LL to R5SS

R3LL-P3 to R5SS-P32

R3LL-P5 to R5SS-P20

5 meters

R3LL to R6SS

R3LL-P6 to R6SS-P19

R3LL-P30 to R6SS-P20

10 meters

R3LL to R7SS

R3LL-P7 to R7SS-P20

R3LL-P36 to R7SS-P32

10 meters

Table 4-29 Leaf Switch Connections for the Fourth Rack in a Seven-Rack System

Switch Pair Connections Cable Length

R4UL to R1SS

R4UL-P3 to R1SS-P31

R4UL-P7 to R1SS-P10

10 meters

R4UL to R2SS

R4UL-P4 to R2SS-P9

R4UL-P6 to R2SS-P10

5 meters

R4UL to R3SS

R4UL-P1 to R3SS-P31

R4UL-P33 to R3SS-P10

5 meters

R4UL to R4SS

R4UL-P32 to R4SS-P10

R4UL-P34 to R4SS-P31

3 meters

R4UL to R5SS

R4UL-P2 to R5SS-P31

R4UL-P5 to R5SS-P10

5 meters

R4UL to R6SS

R4UL-P30 to R6SS-P10

R4UL-P35 to R6SS-P31

5 meters

R4UL to R7SS

R4UL-P31 to R7SS-P10

R4UL-P36 to R7SS-P31

10 meters

R4LL to R1SS

R4LL-P3 to R1SS-P27

R4LL-P5 to R1SS-P17

10 meters

R4LL to R2SS

R4LL-P2 to R2SS-P27

R4LL-P4 to R2SS-P17

5 meters

R4LL to R3SS

R4LL-P31 to R3SS-P17

R4LL-P33 to R3SS-P18

5 meters

R4LL to R4SS

R4LL-P30 to R4SS-P17

R4LL-P34 to R4SS-P27

3 meters

R4LL to R5SS

R4LL-P1 to R5SS-P27

R4LL-P7 to R5SS-P17

5 meters

R4LL to R6SS

R4LL-P6 to R6SS-P17

R4LL-P35 to R6SS-P27

5 meters

R4LL to R7SS

R4LL-P32 to R7SS-P17

R4LL-P36 to R7SS-P27

10 meters

Table 4-30 Leaf Switch Connections for the Fifth Rack in a Seven-Rack System

Switch Pair Connections Cable Length

R5UL to R1SS

R5UL-P3 to R1SS-P24

R5UL-P5 to R1SS-P8

10 meters

R5UL to R2SS

R5UL-P2 to R2SS-P24

R5UL-P4 to R2SS-P8

10 meters

R5UL to R3SS

R5UL-P1 to R3SS-P24

R5UL-P6 to R3SS-P12

5 meters

R5UL to R4SS

R5UL-P31 to R4SS-P7

R5UL-P33 to R4SS-P12

5 meters

R5UL to R5SS

R5UL-P30 to R5SS-P7

R5UL-P34 to R5SS-P25

3 meters

R5UL to R6SS

R5UL-P7 to R6SS-P7

R5UL-P35 to R6SS-P25

5 meters

R5UL to R7SS

R5UL-P32 to R7SS-P7

R5UL-P36 to R7SS-P25

5 meters

R5LL to R1SS

R5LL-P3 to R1SS-P29

R5LL-P7 to R1SS-P18

10 meters

R5LL to R2SS

R5LL-P2 to R2SS-P29

R5LL-P6 to R2SS-P18

10 meters

R5LL to R3SS

R5LL-P31 to R3SS-P16

R5LL-P34 to R3SS-P29

5 meters

R5LL to R4SS

R5LL-P33 to R4SS-P19

R5LL-P35 to R4SS-P29

5 meters

R5LL to R5SS

R5LL-P30 to R5SS-P15

R5LL-P32 to R5SS-P19

3 meters

R5LL to R6SS

R5LL-P1 to R6SS-P29

R5LL-P4 to R6SS-P15

5 meters

R5LL to R7SS

R5LL-P5 to R7SS-P15

R5LL-P36 to R7SS-P29

5 meters

Table 4-31 Leaf Switch Connections for the Sixth Rack in a Seven-Rack System

Switch Pair Connections Cable Length

R6UL to R1SS

R6UL-P3 to R1SS-P23

R6UL-P7 to R1SS-P16

10 meters

R6UL to R2SS

R6UL-P2 to R2SS-P23

R6UL-P4 to R2SS-P16

10 meters

R6UL to R3SS

R6UL-P1 to R3SS-P23

R6UL-P6 to R3SS-P19

10 meters

R6UL to R4SS

R6UL-P31 to R4SS-P15

R6UL-P34 to R4SS-P23

5 meters

R6UL to R5SS

R6UL-P33 to R5SS-P18

R6UL-P35 to R5SS-P23

5 meters

R6UL to R6SS

R6UL-P30 to R6SS-P16

R6UL-P32 to R6SS-P18

3 meters

R6UL to R7SS

R6UL-P5 to R7SS-P16

R6UL-P36 to R7SS-P26

5 meters

R6LL to R1SS

R6LL-P5 to R1SS-P12

R6LL-P7 to R1SS-P15

10 meters

R6LL to R2SS

R6LL-P2 to R2SS-P31

R6LL-P4 to R2SS-P12

10 meters

R6LL to R3SS

R6LL-P3 to R3SS-P28

R6LL-P31 to R3SS-P7

10 meters

R6LL to R4SS

R6LL-P33 to R4SS-P9

R6LL-P34 to R4SS-P24

5 meters

R6LL to R5SS

R6LL-P30 to R5SS-P8

R6LL-P35 to R5SS-P24

5 meters

R6LL to R6SS

R6LL-P32 to R6SS-P8

R6LL-P36 to R6SS-P24

3 meters

R6LL to R7SS

R6LL-P1 to R7SS-P24

R6LL-P6 to R7SS-P8

5 meters

Table 4-32 Leaf Switch Connections for the Seventh Rack in a Seven-Rack System

Switch Pair Connections Cable Length

R7UL to R1SS

R7UL-P3 to R1SS-P30

R7UL-P5 to R1SS-P9

10 meters

R7UL to R2SS

R7UL-P1 to R2SS-P30

R7UL-P7 to R2SS-P7

10 meters

R7UL to R3SS

R7UL-P2 to R3SS-P30

R7UL-P4 to R3SS-P9

10 meters

R7UL to R4SS

R7UL-P6 to R4SS-P8

R7UL-P34 to R4SS-P30

10 meters

R7UL to R5SS

R7UL-P31 to R5SS-P9

R7UL-P35 to R5SS-P30

5 meters

R7UL to R6SS

R7UL-P33 to R6SS-P9

R7UL-P36 to R6SS-P30

5 meters

R7UL to R7SS

R7UL-P30 to R7SS-P9

R7UL-P32 to R7SS-P30

3 meters

R7LL to R1SS

R7LL-P3 to R1SS-P26

R7LL-P5 to R1SS-P19

10 meters

R7LL to R2SS

R7LL-P1 to R2SS-P26

R7LL-P7 to R2SS-P19

10 meters

R7LL to R3SS

R7LL-P2 to R3SS-P26

R7LL-P4 to R3SS-P15

10 meters

R7LL to R4SS

R7LL-P6 to R4SS-P18

R7LL-P34 to R4SS-P26

10 meters

R7LL to R5SS

R7LL-P31 to R5SS-P16

R7LL-P35 to R5SS-P26

5 meters

R7LL to R6SS

R7LL-P33 to R6SS-P12

R7LL-P36 to R6SS-P26

5 meters

R7LL to R7SS

R7LL-P30 to R7SS-P18

R7LL-P32 to R7SS-P19

3 meters

The following table contains all of the RoCE Network Fabric cabling connections from the previous tables. In this table, the connections are sorted by the spine switch port location.

Table 4-33 Seven-Rack System Connections Sorted By The Spine Switch Port Location

Spine Switch Port Leaf Switch Port Cable Length
R1SS-P5 R1UL-P5 3 meters
R1SS-P6 R2UL-P5 5 meters
R1SS-P7 R1UL-P7 3 meters
R1SS-P8 R5UL-P5 10 meters
R1SS-P9 R7UL-P5 10 meters
R1SS-P10 R4UL-P7 10 meters
R1SS-P11 R3UL-P5 5 meters
R1SS-P12 R6LL-P5 10 meters
R1SS-P13 R1LL-P5 3 meters
R1SS-P14 R2LL-P5 5 meters
R1SS-P15 R6LL-P7 10 meters
R1SS-P16 R6UL-P7 10 meters
R1SS-P17 R4LL-P5 10 meters
R1SS-P18 R5LL-P7 10 meters
R1SS-P19 R7LL-P5 10 meters
R1SS-P20 R3LL-P4 5 meters
R1SS-P21 R1LL-P3 3 meters
R1SS-P22 R2LL-P3 5 meters
R1SS-P23 R6UL-P3 10 meters
R1SS-P24 R5UL-P3 10 meters
R1SS-P25 R3UL-P3 5 meters
R1SS-P26 R7LL-P3 10 meters
R1SS-P27 R4LL-P3 10 meters
R1SS-P28 R2UL-P3 5 meters
R1SS-P29 R5LL-P3 10 meters
R1SS-P30 R7UL-P3 10 meters
R1SS-P31 R4UL-P3 10 meters
R1SS-P32 R3LL-P2 5 meters
R2SS-P5 R1UL-P31 5 meters
R2SS-P6 R2UL-P31 3 meters
R2SS-P7 R7UL-P7 10 meters
R2SS-P8 R5UL-P4 10 meters
R2SS-P9 R4UL-P4 5 meters
R2SS-P10 R4UL-P6 5 meters
R2SS-P11 R3UL-P6 5 meters
R2SS-P12 R6LL-P4 10 meters
R2SS-P13 R1LL-P31 5 meters
R2SS-P14 R2LL-P31 3 meters
R2SS-P15 R1LL-P33 5 meters
R2SS-P16 R6UL-P4 10 meters
R2SS-P17 R4LL-P4 5 meters
R2SS-P18 R5LL-P6 10 meters
R2SS-P19 R7LL-P7 10 meters
R2SS-P20 R3LL-P31 5 meters
R2SS-P21 R1UL-P3 5 meters
R2SS-P22 R2LL-P2 3 meters
R2SS-P23 R6UL-P2 10 meters
R2SS-P24 R5UL-P2 10 meters
R2SS-P25 R3UL-P2 5 meters
R2SS-P26 R7LL-P1 10 meters
R2SS-P27 R4LL-P2 5 meters
R2SS-P28 R2UL-P34 3 meters
R2SS-P29 R5LL-P2 10 meters
R2SS-P30 R7UL-P1 10 meters
R2SS-P31 R6LL-P2 10 meters
R2SS-P32 R3LL-P1 5 meters
R3SS-P5 R1UL-P6 5 meters
R3SS-P6 R2UL-P6 5 meters
R3SS-P7 R6LL-P31 10 meters
R3SS-P8 R2UL-P30 5 meters
R3SS-P9 R7UL-P4 10 meters
R3SS-P10 R4UL-P33 5 meters
R3SS-P11 R3UL-P30 3 meters
R3SS-P12 R5UL-P6 5 meters
R3SS-P13 R1LL-P4 5 meters
R3SS-P14 R2LL-P6 5 meters
R3SS-P15 R7LL-P4 10 meters
R3SS-P16 R5LL-P31 5 meters
R3SS-P17 R4LL-P31 5 meters
R3SS-P18 R4LL-P33 5 meters
R3SS-P19 R6UL-P6 10 meters
R3SS-P20 R3LL-P32 3 meters
R3SS-P21 R1LL-P2 5 meters
R3SS-P22 R2LL-P1 5 meters
R3SS-P23 R6UL-P1 10 meters
R3SS-P24 R5UL-P1 5 meters
R3SS-P25 R3UL-P34 3 meters
R3SS-P26 R7LL-P2 10 meters
R3SS-P27 R1UL-P1 5 meters
R3SS-P28 R6LL-P3 10 meters
R3SS-P29 R5LL-P34 5 meters
R3SS-P30 R7UL-P2 10 meters
R3SS-P31 R4UL-P1 5 meters
R3SS-P32 R3LL-P34 3 meters
R4SS-P5 R1UL-P4 10 meters
R4SS-P6 R2UL-P4 5 meters
R4SS-P7 R5UL-P31 5 meters
R4SS-P8 R7UL-P6 10 meters
R4SS-P9 R6LL-P33 5 meters
R4SS-P10 R4UL-P32 3 meters
R4SS-P11 R3UL-P4 5 meters
R4SS-P12 R5UL-P33 5 meters
R4SS-P13 R1LL-P30 10 meters
R4SS-P14 R2LL-P4 5 meters
R4SS-P15 R6UL-P31 5 meters
R4SS-P16 R2LL-P30 5 meters
R4SS-P17 R4LL-P30 3 meters
R4SS-P18 R7LL-P6 10 meters
R4SS-P19 R5LL-P33 5 meters
R4SS-P20 R3LL-P33 5 meters
R4SS-P21 R1LL-P34 10 meters
R4SS-P22 R1UL-P2 10 meters
R4SS-P23 R6UL-P34 5 meters
R4SS-P24 R6LL-P34 5 meters
R4SS-P25 R3UL-P1 5 meters
R4SS-P26 R7LL-P34 10 meters
R4SS-P27 R4LL-P34 3 meters
R4SS-P28 R2UL-P2 5 meters
R4SS-P29 R5LL-P35 5 meters
R4SS-P30 R7UL-P34 10 meters
R4SS-P31 R4UL-P34 3 meters
R4SS-P32 R3LL-P35 5 meters
R5SS-P5 R1UL-P30 10 meters
R5SS-P6 R2UL-P7 10 meters
R5SS-P7 R5UL-P30 3 meters
R5SS-P8 R6LL-P30 5 meters
R5SS-P9 R7UL-P31 5 meters
R5SS-P10 R4UL-P5 5 meters
R5SS-P11 R3UL-P32 5 meters
R5SS-P12 R3UL-P33 5 meters
R5SS-P13 R1LL-P7 10 meters
R5SS-P14 R2LL-P33 10 meters
R5SS-P15 R5LL-P30 3 meters
R5SS-P16 R7LL-P31 5 meters
R5SS-P17 R4LL-P7 5 meters
R5SS-P18 R6UL-P33 5 meters
R5SS-P19 R5LL-P32 3 meters
R5SS-P20 R3LL-P5 5 meters
R5SS-P21 R1LL-P35 10 meters
R5SS-P22 R2LL-P35 10 meters
R5SS-P23 R6UL-P35 5 meters
R5SS-P24 R6LL-P35 5 meters
R5SS-P25 R5UL-P34 3 meters
R5SS-P26 R7LL-P35 5 meters
R5SS-P27 R4LL-P1 5 meters
R5SS-P28 R2UL-P1 10 meters
R5SS-P29 R1UL-P34 10 meters
R5SS-P30 R7UL-P35 5 meters
R5SS-P31 R4UL-P2 5 meters
R5SS-P32 R3LL-P3 5 meters
R6SS-P5 R1UL-P32 10 meters
R6SS-P6 R2UL-P33 10 meters
R6SS-P7 R5UL-P7 5 meters
R6SS-P8 R6LL-P32 3 meters
R6SS-P9 R7UL-P33 5 meters
R6SS-P10 R4UL-P30 5 meters
R6SS-P11 R3UL-P7 10 meters
R6SS-P12 R7LL-P33 5 meters
R6SS-P13 R1LL-P32 10 meters
R6SS-P14 R2LL-P7 10 meters
R6SS-P15 R5LL-P4 5 meters
R6SS-P16 R6UL-P30 3 meters
R6SS-P17 R4LL-P6 5 meters
R6SS-P18 R6UL-P32 3 meters
R6SS-P19 R3LL-P6 10 meters
R6SS-P20 R3LL-P30 10 meters
R6SS-P21 R1LL-P36 10 meters
R6SS-P22 R2LL-P36 10 meters
R6SS-P23 R1UL-P35 10 meters
R6SS-P24 R6LL-P36 3 meters
R6SS-P25 R5UL-P35 5 meters
R6SS-P26 R7LL-P36 5 meters
R6SS-P27 R4LL-P35 5 meters
R6SS-P28 R2UL-P35 10 meters
R6SS-P29 R5LL-P1 5 meters
R6SS-P30 R7UL-P36 5 meters
R6SS-P31 R4UL-P35 5 meters
R6SS-P32 R3UL-P36 10 meters
R7SS-P5 R1UL-P33 10 meters
R7SS-P6 R2UL-P32 10 meters
R7SS-P7 R5UL-P32 5 meters
R7SS-P8 R6LL-P6 5 meters
R7SS-P9 R7UL-P30 3 meters
R7SS-P10 R4UL-P31 10 meters
R7SS-P11 R3UL-P31 10 meters
R7SS-P12 R3UL-P35 10 meters
R7SS-P13 R1LL-P6 10 meters
R7SS-P14 R2LL-P32 10 meters
R7SS-P15 R5LL-P5 5 meters
R7SS-P16 R6UL-P5 5 meters
R7SS-P17 R4LL-P32 10 meters
R7SS-P18 R7LL-P30 3 meters
R7SS-P19 R7LL-P32 3 meters
R7SS-P20 R3LL-P7 10 meters
R7SS-P21 R1LL-P1 10 meters
R7SS-P22 R2LL-P34 10 meters
R7SS-P23 R1UL-P36 10 meters
R7SS-P24 R6LL-P1 5 meters
R7SS-P25 R5UL-P36 5 meters
R7SS-P26 R6UL-P36 5 meters
R7SS-P27 R4LL-P36 10 meters
R7SS-P28 R2UL-P36 10 meters
R7SS-P29 R5LL-P36 5 meters
R7SS-P30 R7UL-P32 3 meters
R7SS-P31 R4UL-P36 10 meters
R7SS-P32 R3LL-P36 10 meters