4 Multi-Rack Cabling Tables for Oracle Engineered System Rack X8M

This appendix contains the tables for multi-rack cabling Oracle Engineered System Rack X8M, which uses a InfiniBand Transport Layer systems based on an InfiniBand Network Layer network.

4.1 Understanding Multi-Rack Cabling for X8M Racks

Up to eight racks can be cabled together without external RDMA Network Fabric switches.

Three RDMA Network Fabric switches are needed when cabling racks together. If any rack does not include a third switch, then order the expansion switch kit. These switches attach to standard RDMA Network Fabric connectors at the end of the cables. The procedures in this section assume the racks are adjacent to each other. If they are not, then longer cables may be required for the connections.

The switch at rack unit 1 (U1) is referred to as the spine switch.

The following switches are referred to as leaf switches:

  • The switches at rack unit 21 (U21) and 23 (U23) in Oracle Engineered System X8M-8.

  • The switches at rack unit 20 (U20) and 22 (U22) in Oracle Engineered System X8M-2.

    Note:

    Oracle Exadata Database Machine X8M racks by default do not include spare cables or a third Cisco Nexus 9336C-FX2 Switch. To extend these racks, you must order cables and a Cisco Nexus 9336C-FX2 Switch. Or simply order the correct cables to extend the rack if the Cisco Nexus 9336C-FX2 Switch is already installed in U1.

In a single rack, the two leaf switches are interconnected using eight connections.

The Oracle Database servers and Oracle Exadata Storage Servers connect to the leaf switches as shown in the following graphic:

Figure 4-1 RDMA Network Fabric Connections Between Database Servers and Storage Servers and Leaf Switches



When connecting up to eight racks together, remove the seven existing inter-switch connections between each leaf switch, as well as the two connections between the leaf switches and the spine switch. From each leaf switch, distribute eight connections over the spine switches in all racks. In multi-rack environments, the leaf switches inside a rack are no longer directly interconnected, as shown in the following graphic:

Figure 4-2 Connections Between Spine Switches and Leaf Switches Across Two Racks

Description of Figure 4-2 follows
Description of "Figure 4-2 Connections Between Spine Switches and Leaf Switches Across Two Racks"

As shown in the preceding graphic, each leaf switch in rack 1 connects to the following switches:

  • Four connections to its internal spine switch
  • Four connections to the spine switch in rack 2

The spine switch in rack 1 connects to the following switches:

  • Eight connections to both internal leaf switches
  • Eight connections to both leaf switches in rack 2

As the number of racks increases from two to eight, the pattern continues as shown in the following graphic:

Figure 4-3 Connections Between Spine Switches and Leaf Switches for up to 8 Racks

Description of Figure 4-3 follows
Description of "Figure 4-3 Connections Between Spine Switches and Leaf Switches for up to 8 Racks"

As shown in the preceding graphic, each leaf switch has eight inter-switch connections distributed over all spine switches. Each spine switch has 16 inter-switch connections distributed over all leaf switches. The leaf switches are not directly interconnected with other leaf switches, and the spine switches are not directly interconnected with the other spine switches.

Note:

Cable lengths are specified for racks 1 through 8.

4.1.1 Preparing to Cable Racks Together

Racks can be added together to increase system capacity and performance. When cabling racks together, note the following:

  • Oracle recommends that the names for the servers include the rack unit number. This helps identify the server during diagnostics.

  • When completing Oracle Exadata Deployment Assistant (OEDA) for the additional rack, you are prompted for SCAN addresses. However, these SCAN addresses are not used because the SCAN address from the original rack are used. Manually remove the new SCAN addresses from the generated installation files.

  • The software owner account names and group names, as well as their identifiers, must match the names and identifiers of the original rack.

  • If the additional grid disks are used with existing disk groups, then ensure the grid disk sizes for the new rack are the same as the original rack.

Note:

Oracle Engineered System Quarter Racks are connected to other racks in the same fashion as larger racks are connected to each other. In other words, a spine switch must exist in the Quarter Rack in order to Multi-rack with other racks.

Perform the following tasks before cabling racks together:

  1. Determine the number of racks that will be cabled together.

  2. Count the spare cables from the kit and any cables.

    Note:

    To extend Oracle Engineered System X8M racks, you must order cables and a Cisco Nexus 9336C-FX2 Switch if not already available.

    When connecting four or more racks, you must purchase additional 10 meter fiber cables with 2x QSFP28 to connect each end. In other words, 2x QSFP SR transceivers are needed for each fiber cable. The following table lists the number of racks, and cables needed to interconnect the racks

    Number of Racks 3 Meter Cables Needed 5 Meter Cables Needed 10 Meter Cables Needed

    2

    16

    16

    0

    3

    18

    30

    0

    4

    16

    40

    8

    5

    20

    42

    18

    6

    24

    46

    26

    7

    28

    44

    40

    8

    16

    52

    60

  3. Gather the following tools:

    • Pen
    • Note pad
    • Flashlight
    • Labels for cables and rack
    • Cable ties (Oracle recommends hook and loop straps)

    See Also:

    Parts for RDMA Network Fabric Switches for information about cable labels
  4. Determine a naming method for the rack prefixes. For example, if the original rack has the prefix dm01, then use the prefix dm02 for the second rack, the prefix dm03 for the third rack, and so on.

  5. Verify the racks have unique host names and IP addresses. All servers interconnected in the racks must have unique names and IP addresses.

    Server names and IP addresses conventions may differ in the following cases:

    • Initial installation of all Oracle Engineered System Full Racks: System address assignments and host names should be complete.

    • New Oracle Engineered System Full Racks are added to an existing cluster: The new rack configuration should require unique host names and IP addresses for the new Oracle Engineered System. The IP addresses on the same subnet cannot conflict with the existing systems.

    • Two existing Oracle Engineered System Full Racks are clustered together: You can assign host names and IP addresses only if Oracle Engineered Systems are already assigned unique host names and IP addresses, or the entire cluster must be reconfigured. The machines must be on the same subnet and not have conflicting IP addresses.

  6. Ensure the IP addresses for the new servers are in the same subnet, and do not overlap with the currently-installed servers.

  7. Ensure the firmware on the original switches are at the same level as the new switches using the sh ver command. If the firmware is not at the same level, then apply a firmware patch.

4.1.2 Cabling Oracle Engineered System Quarter Racks and Eighth Racks

Oracle Engineered System Quarter and Eighth racks are connected to other racks in the same fashion as larger racks are connected to each other. In other words, a spine switch must exist in the Quarter Rack in order to Multi-rack with other racks.

Note:

To extend Oracle Engineered System X8MQuarter and Eighth racks, you must order cables and a Cisco Nexus 9336C-FX2 Switch if one is not already available.
  1. Split each leaf uplink bundle by the number of spines in such a way that the total count adds up to 8. This split is represented in parentheses for each case.

    Example 1: For two racks, take 8 uplinks from each leaf and split evenly by 2. Four uplinks go to rack1-spine, and four uplinks go to rack2-spine ( 4 + 4 )

    Example 2: For three racks, take 8 uplinks from each leaf and split evenly by 3. Three uplinks go to rack1-spine, three uplinks go to rack2-spine, two uplinks go to rack3-spine ( 3 + 3 + 2 ).

  2. Starting from the first available port on a different spine, round-robin the above split scheme for each leaf spine.

    For example, for three racks:

    • leaf-1 starts from rack1-spine for first split, rack2-spine for second split, rack3-spine for third split
    • leaf-2 starts from rack2-spine for first split, rack3-spine for second split, rack1-spine for third split
    • leaf-3 starts from rack3-spine for first split, rack1-spine for second split, rack2-spine for third split
    • and so on...
  3. After walking through all leaf uplinks in each case, you will have used all spine ports between port 5 and 20 inclusive.

4.2 Two-Rack Cabling for X8M Racks

This section provides the cabling tables to connect two Oracle Engineered System Racks together, both of which use InfiniBand Transport Layer systems based on a RoCE Network Layer.

Note:

The following conventions were used in the cabling notation for connecting multiple racks together.

  • The spine switch is in U1 (also referred to as SS) for all racks.

  • The leaf switches are referred to as Lower Leaf (LL) and Upper Leave (UL).

  • The leaf switches are located as follows:

    • At rack unit 21 (U21) and rack unit 23 (U23) in Oracle Engineered System X8M-8

    • At rack unit 20 (U20) and 22 (U22) in Oracle Engineered System X8M-2 racks or Storage Expansion Rack X8M-2

  • The cable lengths shown in the following lists assume that the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cable lengths. Cable lengths up to 100 meters are supported.

  • Only optical cables are supported for lengths greater than 5 meters

The following illustration shows the cable connections for the two spine switches (Rack1-spine and Rack2-spine) when cabling two racks together:

The following table describes the cable connections for the first spine switch (R1-SS) when cabling two racks together:

Table 4-1 Leaf Switch Connections for the First Rack in a Two-Rack System

Leaf Switch Connection Cable Length

R1 UL within Rack 1

R1-UL-P5 to R1-SS-P5

R1-UL-P7 to R1-SS-P7

R1-UL-P4 to R1-SS-P9

R1-UL-P6 to R1-SS-P11

3 meters

R1 UL to Rack 2

R1-UL-P31 to R2-SS-P5

R1-UL-P33 to R2-SS-P7

R1-UL-P30 to R2-SS-P9

R1-UL-P32 to R2-SS-P11

5 meters

R1 LL within Rack 1

R1-LL-P5 to R1-SS-P13

R1-LL-P7 to R1-SS-P13

R1-LL-P4 to R1-SS-P17

R1-LL-P6 to R1-SS-P19

3 meters

R1 LL to Rack 2

R1-LL-P31 to R2-SS-P13

R1-LL-P33 to R2-SS-P15

R1-LL-P30 to R2-SS-P17

R1-LL-P32 to R2-SS-P19

5 meters

The following table describes the cable connections for the second spine switch (R2-SS) when cabling two racks together:

Table 4-2 Leaf Switch Connections for the Second Rack in a Two-Rack System

Leaf Switch Connection Cable Length

R2 UL to Rack 1

R2-UL-P5 to R1-SS-P6

R2-UL-P7 to R1-SS-P8

R2-UL-P4 to R1-SS-P10

R2-UL-P6 to R1-SS-P12

5 meters

R2 UL within Rack 2

R2-UL-P31 to R2-SS-P6

R2-UL-P33 to R2-SS-P8

R2-UL-P30 to R2-SS-P10

R2-UL-P32 to R2-SS-P12

3 meters

R2 LL to Rack 1

R2-LL-P5 to R1-SS-P14

R2-LL-P7 to R1-SS-P16

R2-LL-P4 to R1-SS-P18

R2-LL-P6 to R1-SS-P20

5 meters

R2 LL within Rack 2

R2-LL-P31 to R2-SS-P14

R2-LL-P33 to R2-SS-P16

R2-LL-P30 to R2-SS-P18

R2-LL-P32 to R2-SS-P20

3 meters

4.3 Three-Rack Cabling for X8M Racks

This section provides the cabling tables to connect three Oracle Engineered System Racks together using the InfiniBand Transport Layer systems based on a RoCE Network Layer.

Note:

The following conventions were used in the cabling notation for connecting multiple racks together.

  • The spine switch is in U1 (also referred to as SS) for all racks.

  • The leaf switches are referred to as Lower Leaf (LL) and Upper Leave (UL).

  • The leaf switches are located as follows:

    • At rack unit 21 (U21) and rack unit 23 (U23) in Oracle Engineered System X8M-8

    • At rack unit 20 (U20) and 22 (U22) in Oracle Engineered System X8M-2 racks or Storage Expansion Rack X8M-2

  • The cable lengths shown in the following lists assume that the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cable lengths. Cable lengths up to 100 meters are supported.

  • Only optical cables are supported for lengths greater than 5 meters

The following illustration shows the cable connections for the three spine switches (Rack1-spine, Rack2-spine, and Rack3-spine) when cabling three racks together:



The following table describes the cable connections for the first spine switch (R1-SS) when cabling three racks together:

Table 4-3 Leaf Switch Connections for the First Rack in a Three-Rack System

Leaf Switch Connection Cable Length

R1 UL within Rack 1

R1-UL-P5 to R1-SS-P5

R1-UL-P7 to R1-SS-P7

R1-UL-P4 to R1-SS-P9

3 meters

R1 UL to Rack 2

R1-UL-P31 to R2-SS-P5

R1-UL-P33 to R2-SS-P7

R1-UL-P30 to R2-SS-P9

5 meters

R1 UL to Rack 3

R1-UL-P6 to R3-SS-P5

R1-UL-P32 to R3-SS-P7

5 meters

R1 LL within Rack 1

R1-LL-P5 to R1-SS-P13

R1-LL-P7 to R1-SS-P15

3 meters

R1 LL to Rack 2

R1-LL-P31 to R2-SS-P13

R1-LL-P33 to R2-SS-P15

R1-LL-P30 to R2-SS-P17

5 meters

R1 LL to Rack 3

R1-LL-P4 to R3-SS-P13

R1-LL-P6 to R3-SS-P15

R1-LL-P32 to R3-SS-P17

5 meters

The following table describes the cable connections for the second spine switch (R2-SS) when cabling three racks together:

Table 4-4 Leaf Switch Connections for the Second Rack in a Three-Rack System

Leaf Switch Connection Cable Length

R2 UL to Rack 1

R2-UL-P5 to R1-SS-P6

R2-UL-P7 to R1-SS-P8

R2-UL-P4 to R1-SS-P10

5 meters

R2 UL within Rack 2

R2-UL-P31 to R2-SS-P6

R2-UL-P33 to R2-SS-P8

3 meters

R2 UL to Rack 3

R2-UL-P6 to R3-SS-P6

R2-UL-P30 to R3-SS-P8

R2-UL-P32 to R3-SS-P10

5 meters

R2 LL to Rack 1

R2-LL-P5 to R1-SS-P14

R2-LL-P7 to R1-SS-P16

R2-LL-P4 to R1-SS-P18

5 meters

R2 LL within Rack 2

R2-LL-P31 to R2-SS-P14

R2-LL-P33 to R2-SS-P16

R2-LL-P30 to R2-SS-P18

3 meters

R2 LL to Rack 3

R2-LL-P6 to R3-SS-P14

R2-LL-P32 to R3-SS-P16

5 meters

The following table describes the cable connections for the third spine switch (R3-SS) when cabling three racks together:

Table 4-5 Leaf Switch Connections for the Third Rack in a Three-Rack System

Leaf Switch Connection Cable Length

R3 UL to Rack 1

R3-UL-P5 to R1-SS-P11

R3-UL-P7 to R1-SS-P12

5 meters

R3 UL to Rack 2

R3-UL-P4 to R2-SS-P10

R3-UL-P6 to R2-SS-P11

R3-UL-P31 to R2-SS-P12

5 meters

R3 UL within Rack 3

R3-UL-P33 to R3-SS-P9

R3-UL-P30 to R3-SS-P11

R3-UL-P32 to R3-SS-P12

3 meters

R3 LL to Rack 1

R3-LL-P5 to R1-SS-P17

R3-LL-P7 to R1-SS-P19

R3-LL-P4 to R1-SS-P20

5 meters

R3 LL to Rack 2

R3-LL-P6 to R2-SS-P19

R3-LL-P31 to R2-SS-P20

5 meters

R3 LL within Rack 3

R3-LL-P33 to R3-SS-P18

R3-LL-P30 to R3-SS-P19

R3-LL-P32 to R3-SS-P20

3 meters

4.4 Four-Rack Cabling for X8M Racks

This section provides the cabling tables to connect four Oracle Engineered System Racks together, all of which use InfiniBand Transport Layer systems based on a RoCE Network Layer.

Note:

The following conventions were used in the cabling notation for connecting multiple racks together.

  • The spine switch is in U1 (also referred to as SS) for all racks.

  • The leaf switches are referred to as Lower Leaf (LL) and Upper Leave (UL).

  • The leaf switches are located as follows:

    • At rack unit 21 (U21) and rack unit 23 (U23) in Oracle Engineered System X8M-8

    • At rack unit 20 (U20) and 22 (U22) in Oracle Engineered System X8M-2 racks or Storage Expansion Rack X8M-2

  • The cable lengths shown in the following lists assume that the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cable lengths. Cable lengths up to 100 meters are supported.

  • Only optical cables are supported for lengths greater than 5 meters

The following illustration shows the cable connections for the four spine switches (Rack1-spine, Rack2-spine, Rack3-spine, and Rack4-spine) when cabling two racks together:



The following table describes the cable connections for the first spine switch (R1-SS) when cabling four racks together:

Table 4-6 Leaf Switch Connections for the First Rack in a Four-Rack System

Leaf Switch Connection Cable Length

R1 UL within Rack 1

R1-UL-P5 to R1-SS-P5

R1-UL-P7 to R1-SS-P7

3 meters

R1 UL to Rack 2

R1-UL-P31 to R2-SS-P5

R1-UL-P33 to R2-SS-P7

5 meters

R1 UL to Rack 3

R1-UL-P6 to R3-SS-P5

R1-UL-P32 to R3-SS-P7

5 meters

R1 UL to Rack 4

R1-UL-P4 to R4-SS-P5

R1-UL-P30 to R4-SS-P7

10 meters

R1 LL within Rack 1

R1-LL-P5 to R1-SS-P13

R1-LL-P7 to R1-SS-P15

3 meters

R1 LL to Rack 2

R1-LL-P31 to R2-SS-P13

R1-LL-P33 to R2-SS-P15

5 meters

R1 LL to Rack 3

R1-LL-P4 to R3-SS-P13

R1-LL-P6 to R3-SS-P15

5 meters

R1 LL to Rack 4

R1-LL-P30 to R4-SS-P13

R1-LL-P32 to R4-SS-P15

10 meters

The following table describes the cable connections for the second spine switch (R2-SS) when cabling four full racks together:

Table 4-7 Leaf Switch Connections for the Second Rack in a Four-Rack System

Leaf Switch Connection Cable Length

R2 UL to Rack 1

R2-UL-P5 to R1-SS-P6

R2-UL-P7 to R1-SS-P8

5 meters

R2 UL within Rack 2

R2-UL-P31 to R2-SS-P6

R2-UL-P33 to R2-SS-P8

3 meters

R2 UL to Rack 3

R2-UL-P6 to R3-SS-P6

R2-UL-P30 to R3-SS-P8

5 meters

R2 UL to Rack 4

R2-UL-P4 to R4-SS-P6

R2-UL-P32 to R4-SS-P8

5 meters

R2 LL to Rack 1

R2-LL-P5 to R1-SS-P14

R2-LL-P7 to R1-SS-P16

5 meters

R2 LL within Rack 2

R2-LL-P31 to R2-SS-P14

R2-LL-P33 to R2-SS-P16

3 meters

R2 LL to Rack 3

R2-LL-P6 to R3-SS-P14

R2-LL-P32 to R3-SS-P16

5 meters

R2 LL to Rack 4

R2-LL-P4 to R4-SS-P14

R2-LL-P30 to R4-SS-P16

5 meters

The following table describes the cable connections for the third spine switch (R3-SS) when cabling four full racks together:

Table 4-8 Leaf Switch Connections for the Third Rack in a Four-Rack System

Leaf Switch Connection Cable Length

R3 UL to Rack 1

R3-UL-P5 to R1-SS-P11

R3-UL-P7 to R1-SS-P12

5 meters

R3 UL to Rack 2

R3-UL-P6 to R2-SS-P11

R3-UL-P31 to R2-SS-P12

5 meters

R3 UL within Rack 3

R3-UL-P30 to R3-SS-P11

R3-UL-P32 to R3-SS-P12

3 meters

R3 UL to Rack 4

R3-UL-P4 to R4-SS-P11

R3-UL-P33 to R4-SS-P12

5 meters

R3 LL to Rack 1

R3-LL-P7 to R1-SS-P19

R3-LL-P4 to R1-SS-P20

5 meters

R3 LL to Rack 2

R3-LL-P6 to R2-SS-P19

R3-LL-P31 to R2-SS-P20

5 meters

R3 LL within Rack 3

R3-LL-P30 to R3-SS-P19

R3-LL-P32 to R3-SS-P20

3 meters

R3 LL to Rack 4

R3-LL-P5 to R4-SS-P19

R3-LL-P33 to R4-SS-P20

5 meters

The following table describes the cable connections for the fourth spine switch (R4-SS) when cabling four full racks together:

Table 4-9 Leaf Switch Connections for the Fourth Rack in a Four-Rack System

Leaf Switch Connection Cable Length

R4 UL to Rack 1

R4-UL-P5 to R1-SS-P9

R4-UL-P7 to R1-SS-P10

10 meters

R4 UL to Rack 2

R4-UL-P4 to R2-SS-P9

R4-UL-P6 to R2-SS-P10

5 meters

R4 UL to Rack 3

R4-UL-P31 to R3-SS-P9

R4-UL-P33 to R3-SS-P10

5 meters

R4 UL within Rack 4

R4-UL-P30 to R4-SS-P9

R4-UL-P32 to R4-SS-P10

3 meters

R4 LL to Rack 1

R4-LL-P5 to R1-SS-P17

R4-LL-P7 to R1-SS-P18

10 meters

R4 LL to Rack 2

R4-LL-P4 to R2-SS-P17

R4-LL-P6 to R2-SS-P18

5 meters

R4 LL to Rack 3

R4-LL-P31 to R3-SS-P17

R4-LL-P33 to R3-SS-P18

5 meters

R4 LL within Rack 4

R4-LL-P30 to R4-SS-P17

R4-LL-P32 to R4-SS-P18

3 meters

4.5 Five-Rack Cabling for X8M Racks

This section provides the cabling tables to connect five (5) Oracle Engineered System Racks together, all of which use InfiniBand Transport Layer systems based on a RoCE Network Layer.

Note:

The following conventions were used in the cabling notation for connecting multiple racks together.

  • The spine switch is in U1 (also referred to as SS) for all racks.

  • The leaf switches are referred to as Lower Leaf (LL) and Upper Leave (UL).

  • The leaf switches are located as follows:

    • At rack unit 21 (U21) and rack unit 23 (U23) in Oracle Engineered System X8M-8

    • At rack unit 20 (U20) and 22 (U22) in Oracle Engineered System X8M-2 racks or Storage Expansion Rack X8M-2

  • The cable lengths shown in the following lists assume that the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cable lengths. Cable lengths up to 100 meters are supported.

  • Only optical cables are supported for lengths greater than 5 meters

The following illustration shows the cable connections for the five spine switches when cabling five racks together:



The following table describes the cable connections for the first spine switch (R1-SS) when cabling five racks together:

Table 4-10 Leaf Switch Connections for the First Rack in a Five-Rack System

Leaf Switch Connection Cable Length

R1 UL within Rack 1

R1-UL-P5 to R1-SS-P5

R1-UL-P7 to R1-SS-P7

3 meters

R1 UL to Rack 2

R1-UL-P31 to R2-SS-P5

R1-UL-P33 to R2-SS-P7

5 meters

R1 UL to Rack 3

R1-UL-P6 to R3-SS-P5

R1-UL-P32 to R3-SS-P7

5 meters

R1 UL to Rack 4

R1-UL-P4 to R4-SS-P5

10 meters

R1 UL to Rack 5

R1-UL-P30 to R5-SS-P5

10 meters

R1 LL within Rack 1

R1-LL-P5 to R1-SS-P13

3 meters

R1 LL to Rack 2

R1-LL-P31 to R2-SS-P13

R1-LL-P33 to R2-SS-P15

5 meters

R1 LL to Rack 3

R1-LL-P4 to R3-SS-P13

R1-LL-P6 to R3-SS-P15

5 meters

R1 LL to Rack 4

R1-LL-P30 to R4-SS-P13

R1-LL-P32 to R4-SS-P15

10 meters

R1 LL to Rack 5

R1-LL-P7 to R5-SS-P13

10 meters

The following table describes the cable connections for the second spine switch (R2-SS) when cabling five full racks together:

Table 4-11 Leaf Switch Connections for the Second Rack in a Five-Rack System

Leaf Switch Connection Cable Length

R2 UL to Rack 1

R2-UL-P5 to R1-SS-P6

5 meters

R2 UL within Rack 2

R2-UL-P31 to R2-SS-P6

3 meters

R2 UL to Rack 3

R2-UL-P6 to R3-SS-P6

R2-UL-P30 to R3-SS-P8

5 meters

R2 UL to Rack 4

R2-UL-P4 to R4-SS-P6

R2-UL-P32 to R4-SS-P8

5 meters

R2 UL to Rack 5

R2-UL-P7 to R5-SS-P6

R2-UL-P33 to R5-SS-P8

10 meters

R2 LL to Rack 1

R2-LL-P5 to R1-SS-P14

R2-LL-P7 to R1-SS-P16

5 meters

R2 LL within Rack 2

R2-LL-P31 to R2-SS-P14

3 meters

R2 LL to Rack 3

R2-LL-P6 to R3-SS-P14

5 meters

R2 LL to Rack 4

R2-LL-P4 to R4-SS-P14

R2-LL-P30 to R4-SS-P16

5 meters

R2 LL to Rack 5

R2-LL-P33 to R5-SS-P14

R2-LL-P32 to R5-SS-P16

10 meters

The following table describes the cable connections for the third spine switch (R3-SS) when cabling five full racks together:

Table 4-12 Leaf Switch Connections for the Third Rack in a Five-Rack System

Leaf Switch Connection Cable Length

R3 UL to Rack 1

R3-UL-P5 to R1-SS-P11

R3-UL-P7 to R1-SS-P12

5 meters

R3 UL to Rack 2

R3-UL-P6 to R2-SS-P11

R3-UL-P31 to R2-SS-P12

5 meters

R3 UL within Rack 3

R3-UL-P30 to R3-SS-P11

3 meters

R3 UL to Rack 4

R3-UL-P4 to R4-SS-P11

5 meters

R3 UL to Rack 5

R3-UL-P32 to R5-SS-P11

R3-UL-P33 to R5-SS-P12

5 meters

R3 LL to Rack 1

R3-LL-P7 to R1-SS-P19

R3-LL-P4 to R1-SS-P20

5 meters

R3 LL to Rack 2

R3-LL-P6 to R2-SS-P19

R3-LL-P31 to R2-SS-P20

5 meters

R3 LL within Rack 3

R3-LL-P30 to R3-SS-P19

R3-LL-P32 to R3-SS-P20

3 meters

R3 LL to Rack 4

R3-LL-P33 to R4-SS-P20

5 meters

R3 LL to Rack 5

R3-LL-P5 to R5-SS-P20

5 meters

The following table describes the cable connections for the fourth spine switch (R4-SS) when cabling five full racks together:

Table 4-13 Leaf Switch Connections for the Fourth Rack in a Five-Rack System

Leaf Switch Connection Cable Length

R4 UL to Rack 1

R4-UL-P7 to R1-SS-P10

10 meters

R4 UL to Rack 2

R4-UL-P4 to R2-SS-P9

R4-UL-P6 to R2-SS-P10

5 meters

R4 UL to Rack 3

R4-UL-P31 to R3-SS-P9

R4-UL-P33 to R3-SS-P10

5 meters

R4 UL within Rack 4

R4-UL-P30 to R4-SS-P9

R4-UL-P32 to R4-SS-P10

3 meters

R4 UL to Rack 5

R4-UL-P5 to R5-SS-P10

5 meters

R4 LL to Rack 1

R4-LL-P5 to R1-SS-P17

10 meters

R4 LL to Rack 2

R4-LL-P4 to R2-SS-P17

5 meters

R4 LL to Rack 3

R4-LL-P31 to R3-SS-P17

R4-LL-P33 to R3-SS-P18

5 meters

R4 LL within Rack 4

R4-LL-P30 to R4-SS-P17

R4-LL-P32 to R4-SS-P18

3 meters

R4 LL to Rack 5

R4-LL-P7 to R5-SS-P17

R4-LL-P6 to R5-SS-P18

5 meters

The following table describes the cable connections for the fifth spine switch (R5-SS) when cabling five full racks together:

Table 4-14 Leaf Switch Connections for the Fifth Rack in a Five-Rack System

Leaf Switch Connection Cable Length

R5 UL to Rack 1

R5-UL-P5 to R1-SS-P8

R5-UL-P7 to R1-SS-P9

10 meters

R5 UL to Rack 2

R5-UL-P4 to R2-SS-P8

10 meters

R5 UL to Rack 3

R5-UL-P6 to R3-SS-P12

5 meters

R5 UL to Rack 4

R5-UL-P31 to R4-SS-P7

R5-UL-P33 to R4-SS-P12

5 meters

R5 UL within Rack 5

R5-UL-P30 to R5-SS-P7

R5-UL-P32 to R5-SS-P9

3 meters

R5 LL to Rack 1

R5-UL-P5 to R1-SS-P15

R5-UL-P7 to R1-SS-P18

10 meters

R5 LL to Rack 2

R5-UL-P4 to R2-SS-P16

R5-UL-P6 to R2-SS-P18

10 meters

R5 LL to Rack 3

R5-UL-P31 to R3-SS-P16

5 meters

R5 LL to Rack 4

R5-UL-P33 to R4-SS-P19

5 meters

R5 LL within Rack 5

R5-UL-P30 to R5-SS-P15

R5-UL-P32 to R5-SS-P19

3 meters

4.6 Six-Rack Cabling forX8M Racks

This section provides the cabling tables to connect six (6) Oracle Engineered System Racks together, all of which use InfiniBand Transport Layer systems based on a RoCE Network Layer.

Note:

The following conventions were used in the cabling notation for connecting multiple racks together.

  • The spine switch is in U1 (also referred to as SS) for all racks.

  • The leaf switches are referred to as Lower Leaf (LL) and Upper Leave (UL).

  • The leaf switches are located as follows:

    • At rack unit 21 (U21) and rack unit 23 (U23) in Oracle Engineered System X8M-8

    • At rack unit 20 (U20) and 22 (U22) in Oracle Engineered System X8M-2 racks or Storage Expansion Rack X8M-2

  • The cable lengths shown in the following lists assume that the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cable lengths. Cable lengths up to 100 meters are supported.

  • Only optical cables are supported for lengths greater than 5 meters

The following illustration shows the cable connections for the 6 spine switches when cabling six racks together:



The following table describes the cable connections for the first spine switch (R1-SS) when cabling six racks together:

Table 4-15 Leaf Switch Connections for the First Rack in a Six-Rack System

Leaf Switch Connection Cable Length

R1 UL within Rack 1

R1-UL-P5 to R1-SS-P5

R1-UL-P7 to R1-SS-P7

3 meters

R1 UL to Rack 2

R1-UL-P31 to R2-SS-P5

R1-UL-P33 to R2-SS-P7

5 meters

R1 UL to Rack 3

R1-UL-P6 to R3-SS-P5

5 meters

R1 UL to Rack 4

R1-UL-P4 to R4-SS-P5

10 meters

R1 UL to Rack 5

R1-UL-P30 to R5-SS-P5

10 meters

R1 UL to Rack 6

R1-UL-P32 to R6-SS-P5

10 meters

R1 LL within Rack 1

R1-LL-P5 to R1-SS-P13

3 meters

R1 LL to Rack 2

R1-LL-P31 to R2-SS-P13

R1-LL-P33 to R2-SS-P15

5 meters

R1 LL to Rack 3

R1-LL-P4 to R3-SS-P13

R1-LL-P6 to R3-SS-P15

5 meters

R1 LL to Rack 4

R1-LL-P30 to R4-SS-P13

10 meters

R1 LL to Rack 5

R1-LL-P7 to R5-SS-P13

10 meters

R1 LL to Rack 6

R1-LL-P32 to R6-SS-P13

10 meters

The following table describes the cable connections for the second spine switch (R2-SS) when cabling six full racks together:

Table 4-16 Leaf Switch Connections for the Second Rack in a Six-Rack System

Leaf Switch Connection Cable Length

R2 UL to Rack 1

R2-UL-P5 to R1-SS-P6

5 meters

R2 UL within Rack 2

R2-UL-P31 to R2-SS-P6

3 meters

R2 UL to Rack 3

R2-UL-P6 to R3-SS-P6

R2-UL-P30 to R3-SS-P8

5 meters

R2 UL to Rack 4

R2-UL-P4 to R4-SS-P6

R2-UL-P32 to R4-SS-P8

5 meters

R2 UL to Rack 5

R2-UL-P7 to R5-SS-P6

10 meters

R2 UL to Rack 6

R2-UL-P33 to R6-SS-P6

10 meters

R2 LL to Rack 1

R2-LL-P5 to R1-SS-P14

5 meters

R2 LL within Rack 2

R2-LL-P31 to R2-SS-P14

3 meters

R2 LL to Rack 3

R2-LL-P6 to R3-SS-P14

5 meters

R2 LL to Rack 4

R2-LL-P4 to R4-SS-P14

R2-LL-P30 to R4-SS-P16

5 meters

R2 LL to Rack 5

R2-LL-P33 to R5-SS-P14

R2-LL-P32 to R5-SS-P16

10 meters

R2 LL to Rack 6

R2-LL-P7 to R6-SS-P14

10 meters

The following table describes the cable connections for the third spine switch (R3-SS) when cabling six full racks together:

Table 4-17 Leaf Switch Connections for the Third Rack in a Six-Rack System

Leaf Switch Connection Cable Length

R3 UL to Rack 1

R3-UL-P5 to R1-SS-P11

5 meters

R3 UL to Rack 2

R3-UL-P6 to R2-SS-P11

5 meters

R3 UL within Rack 3

R3-UL-P30 to R3-SS-P11

3 meters

R3 UL to Rack 4

R3-UL-P4 to R4-SS-P11

5 meters

R3 UL to Rack 5

R3-UL-P32 to R5-SS-P11

R3-UL-P33 to R5-SS-P12

5 meters

R3 UL to Rack 6

R3-UL-P7 to R6-SS-P11

R3-UL-P31 to R6-SS-P12

10 meters

R3 LL to Rack 1

R3-LL-P7 to R1-SS-P19

R3-LL-P4 to R1-SS-P20

5 meters

R3 LL to Rack 2

R3-LL-P31 to R2-SS-P20

5 meters

R3 LL within Rack 3

R3-LL-P32 to R3-SS-P20

3 meters

R3 LL to Rack 4

R3-LL-P33 to R4-SS-P20

5 meters

R3 LL to Rack 5

R3-LL-P5 to R5-SS-P20

5 meters

R3 LL to Rack 6

R3-LL-P6 to R6-SS-P19

R3-LL-P30 to R6-SS-P20

10 meters

The following table describes the cable connections for the fourth spine switch (R4-SS) when cabling six full racks together:

Table 4-18 Leaf Switch Connections for the Fourth Rack in a Six-Rack System

Leaf Switch Connection Cable Length

R4 UL to Rack 1

R4-UL-P7 to R1-SS-P10

10 meters

R4 UL to Rack 2

R4-UL-P4 to R2-SS-P9

R4-UL-P6 to R2-SS-P10

5 meters

R4 UL to Rack 3

R4-UL-P31 to R3-SS-P9

R4-UL-P33 to R3-SS-P10

5 meters

R4 UL within Rack 4

R4-UL-P32 to R4-SS-P10

3 meters

R4 UL to Rack 5

R4-UL-P5 to R5-SS-P10

5 meters

R4 UL to Rack 6

R4-UL-P30 to R6-SS-P10

5 meters

R4 LL to Rack 1

R4-LL-P5 to R1-SS-P17

10 meters

R4 LL to Rack 2

R4-LL-P4 to R2-SS-P17

5 meters

R4 LL to Rack 3

R4-LL-P31 to R3-SS-P17

R4-LL-P33 to R3-SS-P18

5 meters

R4 LL within Rack 4

R4-LL-P30 to R4-SS-P17

R4-LL-P32 to R4-SS-P18

3 meters

R4 LL to Rack 5

R4-LL-P7 to R5-SS-P17

5 meters

R4 LL to Rack 6

R4-LL-P6 to R6-SS-P17

5 meters

The following table describes the cable connections for the fifth spine switch (R5-SS) when cabling six full racks together:

Table 4-19 Leaf Switch Connections for the Fifth Rack in a Six-Rack System

Leaf Switch Connection Cable Length

R5 UL to Rack 1

R5-UL-P5 to R1-SS-P8

10 meters

R5 UL to Rack 2

R5-UL-P4 to R2-SS-P8

10 meters

R5 UL to Rack 3

R5-UL-P6 to R3-SS-P12

5 meters

R5 UL to Rack 4

R5-UL-P31 to R4-SS-P7

R5-UL-P33 to R4-SS-P12

5 meters

R5 UL within Rack 5

R5-UL-P30 to R5-SS-P7

R5-UL-P32 to R5-SS-P9

3 meters

R5 UL to Rack 6

R5-UL-P7 to R6-SS-P7

5 meters

R5 LL to Rack 1

R5-LL-P7 to R1-SS-P18

10 meters

R5 LL to Rack 2

R5-LL-P6 to R2-SS-P18

10 meters

R5 LL to Rack 3

R5-LL-P31 to R3-SS-P16

5 meters

R5 LL to Rack 4

R5-LL-P33 to R4-SS-P19

5 meters

R5 LL within Rack 5

R5-LL-P30 to R5-SS-P15

R5-LL-P32 to R5-SS-P19

3 meters

R5 LL to Rack 6

R5-LL-P5 to R6-SS-P9

R5-LL-P4 to R6-SS-P15

5 meters

The following table describes the cable connections for the sixth spine switch (R6-SS) when cabling six full racks together:

Table 4-20 Leaf Switch Connections for the Sixth Rack in a Six-Rack System

Leaf Switch Connection Cable Length

R6 UL to Rack 1

R6-UL-P5 to R1-SS-P9

R6-UL-P7 to R1-SS-P16

10 meters

R6 UL to Rack 2

R6-UL-P4 to R2-SS-P16

10 meters

R6 UL to Rack 3

R6-UL-P6 to R3-SS-P19

10 meters

R6 UL to Rack 4

R6-UL-P31 to R4-SS-P15

5 meters

R6 UL to Rack 5

R6-UL-P33 to R5-SS-P18

5 meters

R6 UL within Rack 6

R6-UL-P30 to R6-SS-P16

R6-UL-P32 to R6-SS-P18

3 meters

R6 LL to Rack 1

R6-LL-P5 to R1-SS-P12

R6-LL-P7 to R1-SS-P15

10 meters

R6 LL to Rack 2

R6-LL-P4 to R2-SS-P12

R6-LL-P6 to R2-SS-P19

10 meters

R6 LL to Rack 3

R6-LL-P31 to R3-SS-P7

10 meters

R6 LL to Rack 4

R6-LL-P33 to R4-SS-P9

5 meters

R6 LL to Rack 5

R6-LL-P30 to R5-SS-P8

5 meters

R6 LL within Rack 6

R6-LL-P32 to R6-SS-P8

3 meters

4.7 Seven-Rack Cabling for X8M Racks

This section provides the cabling tables to connect seven (7) Oracle Engineered System Racks together, all of which use InfiniBand Transport Layer systems based on a RoCE Network Layer.

Note:

The following conventions were used in the cabling notation for connecting multiple racks together.

  • The spine switch is in U1 (also referred to as SS) for all racks.

  • The leaf switches are referred to as Lower Leaf (LL) and Upper Leave (UL).

  • The leaf switches are located as follows:

    • At rack unit 21 (U21) and rack unit 23 (U23) in Oracle Engineered System X8M-8

    • At rack unit 20 (U20) and 22 (U22) in Oracle Engineered System X8M-2 racks or Storage Expansion Rack X8M-2

  • The cable lengths shown in the following lists assume that the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cable lengths. Cable lengths up to 100 meters are supported.

  • Only optical cables are supported for lengths greater than 5 meters

The following illustration shows the cable connections for the 7 spine switches when cabling seven racks together:



The following table describes the cable connections for the first spine switch (R1-SS) when cabling seven racks together:

Table 4-21 Leaf Switch Connections for the First Rack in a Seven-Rack System

Leaf Switch Connection Cable Length

R1 UL within Rack 1

R1-UL-P5 to R1-SS-P5

R1-UL-P7 to R1-SS-P7

3 meters

R1 UL to Rack 2

R1-UL-P31 to R2-SS-P5

5 meters

R1 UL to Rack 3

R1-UL-P6 to R3-SS-P5

5 meters

R1 UL to Rack 4

R1-UL-P4 to R4-SS-P5

10 meters

R1 UL to Rack 5

R1-UL-P30 to R5-SS-P5

10 meters

R1 UL to Rack 6

R1-UL-P32 to R6-SS-P5

10 meters

R1 UL to Rack 7

R1-UL-P33 to R7-SS-P5

10 meters

R1 LL within Rack 1

R1-LL-P5 to R1-SS-P13

3 meters

R1 LL to Rack 2

R1-LL-P31 to R2-SS-P13

R1-LL-P33 to R2-SS-P15

5 meters

R1 LL to Rack 3

R1-LL-P4 to R3-SS-P13

5 meters

R1 LL to Rack 4

R1-LL-P30 to R4-SS-P13

10 meters

R1 LL to Rack 5

R1-LL-P7 to R5-SS-P13

10 meters

R1 LL to Rack 6

R1-LL-P32 to R6-SS-P13

10 meters

R1 LL to Rack 7

R1-LL-P6 to R7-SS-P13

10 meters

The following table describes the cable connections for the second spine switch (R2-SS) when cabling seven full racks together:

Table 4-22 Leaf Switch Connections for the Second Rack in a Seven-Rack System

Leaf Switch Connection Cable Length

R2 UL to Rack 1

R2-UL-P5 to R1-SS-P6

5 meters

R2 UL within Rack 2

R2-UL-P31 to R2-SS-P6

3 meters

R2 UL to Rack 3

R2-UL-P6 to R3-SS-P6

R2-UL-P30 to R3-SS-P8

5 meters

R2 UL to Rack 4

R2-UL-P4 to R4-SS-P6

5 meters

R2 UL to Rack 5

R2-UL-P7 to R5-SS-P6

10 meters

R2 UL to Rack 6

R2-UL-P33 to R6-SS-P6

10 meters

R2 UL to Rack 7

R2-UL-P32 to R7-SS-P6

10 meters

R2 LL to Rack 1

R2-LL-P5 to R1-SS-P14

5 meters

R2 LL within Rack 2

R2-LL-P31 to R2-SS-P14

3 meters

R2 LL to Rack 3

R2-LL-P6 to R3-SS-P14

5 meters

R2 LL to Rack 4

R2-LL-P4 to R4-SS-P14

R2-LL-P30 to R4-SS-P16

5 meters

R2 LL to Rack 5

R2-LL-P33 to R5-SS-P14

10 meters

R2 LL to Rack 6

R2-LL-P7 to R6-SS-P14

10 meters

R2 LL to Rack 7

R2-LL-P32 to R7-SS-P14

10 meters

The following table describes the cable connections for the third spine switch (R3-SS) when cabling seven full racks together:

Table 4-23 Leaf Switch Connections for the Third Rack in a Seven-Rack System

Leaf Switch Connection Cable Length

R3 UL to Rack 1

R3-UL-P5 to R1-SS-P11

5 meters

R3 UL to Rack 2

R3-UL-P6 to R2-SS-P11

5 meters

R3 UL within Rack 3

R3-UL-P30 to R3-SS-P11

3 meters

R3 UL to Rack 4

R3-UL-P4 to R4-SS-P11

5 meters

R3 UL to Rack 5

R3-UL-P32 to R5-SS-P11

R3-UL-P33 to R5-SS-P12

5 meters

R3 UL to Rack 6

R3-UL-P7 to R6-SS-P11

10 meters

R3 UL to Rack 7

R3-UL-P31 to R7-SS-P11

10 meters

R3 LL to Rack 1

R3-LL-P4 to R1-SS-P20

5 meters

R3 LL to Rack 2

R3-LL-P31 to R2-SS-P20

5 meters

R3 LL within Rack 3

R3-LL-P32 to R3-SS-P20

3 meters

R3 LL to Rack 4

R3-LL-P33 to R4-SS-P20

5 meters

R3 LL to Rack 5

R3-LL-P5 to R5-SS-P20

5 meters

R3 LL to Rack 6

R3-LL-P6 to R6-SS-P19

R3-LL-P30 to R6-SS-P20

10 meters

R3 LL to Rack 7

R3-LL-P7 to R7-SS-P20

10 meters

The following table describes the cable connections for the fourth spine switch (R4-SS) when cabling seven full racks together:

Table 4-24 Leaf Switch Connections for the Fourth Rack in a Seven-Rack System

Leaf Switch Connection Cable Length

R4 UL to Rack 1

R4-UL-P7 to R1-SS-P10

10 meters

R4 UL to Rack 2

R4-UL-P4 to R2-SS-P9

R4-UL-P6 to R2-SS-P10

5 meters

R4 UL to Rack 3

R4-UL-P33 to R3-SS-P10

5 meters

R4 UL within Rack 4

R4-UL-P32 to R4-SS-P10

3 meters

R4 UL to Rack 5

R4-UL-P5 to R5-SS-P10

5 meters

R4 UL to Rack 6

R4-UL-P30 to R6-SS-P10

5 meters

R4 UL to Rack 7

R4-UL-P31 to R7-SS-P10

10 meters

R4 LL to Rack 1

R4-LL-P5 to R1-SS-P17

10 meters

R4 LL to Rack 2

R4-LL-P4 to R2-SS-P17

5 meters

R4 LL to Rack 3

R4-LL-P31 to R3-SS-P17

R4-LL-P33 to R3-SS-P18

5 meters

R4 LL within Rack 4

R4-LL-P30 to R4-SS-P17

3 meters

R4 LL to Rack 5

R4-LL-P7 to R5-SS-P17

5 meters

R4 LL to Rack 6

R4-LL-P6 to R6-SS-P17

5 meters

R4 LL to Rack 7

R4-LL-P32 to R7-SS-P17

10 meters

The following table describes the cable connections for the fifth spine switch (R5-SS) when cabling seven full racks together:

Table 4-25 Leaf Switch Connections for the Fifth Rack in a Seven-Rack System

Leaf Switch Connection Cable Length

R5 UL to Rack 1

R5-UL-P5 to R1-SS-P8

10 meters

R5 UL to Rack 2

R5-UL-P4 to R2-SS-P8

10 meters

R5 UL to Rack 3

R5-UL-P6 to R3-SS-P12

5 meters

R5 UL to Rack 4

R5-UL-P31 to R4-SS-P7

R5-UL-P33 to R4-SS-P12

5 meters

R5 UL within Rack 5

R5-UL-P30 to R5-SS-P7

3 meters

R5 UL to Rack 6

R5-UL-P7 to R6-SS-P7

5 meters

R5 UL to Rack 7

R5-UL-P32 to R7-SS-P7

5 meters

R5 LL to Rack 1

R5-LL-P7 to R1-SS-P18

10 meters

R5 LL to Rack 2

R5-LL-P6 to R2-SS-P18

10 meters

R5 LL to Rack 3

R5-LL-P31 to R3-SS-P16

5 meters

R5 LL to Rack 4

R5-LL-P33 to R4-SS-P19

5 meters

R5 LL within Rack 5

R5-LL-P30 to R5-SS-P15

R5-LL-P32 to R5-SS-P19

3 meters

R5 LL to Rack 6

R5-LL-P4 to R6-SS-P15

5 meters

R5 LL to Rack 7

R5-LL-P5 to R7-SS-P15

5 meters

The following table describes the cable connections for the sixth spine switch (R6-SS) when cabling seven full racks together:

Table 4-26 Leaf Switch Connections for the Sixth Rack in a Seven-Rack System

Leaf Switch Connection Cable Length

R6 UL to Rack 1

R6-UL-P7 to R1-SS-P16

10 meters

R6 UL to Rack 2

R6-UL-P4 to R2-SS-P16

10 meters

R6 UL to Rack 3

R6-UL-P6 to R3-SS-P19

10 meters

R6 UL to Rack 4

R6-UL-P31 to R4-SS-P15

5 meters

R6 UL to Rack 5

R6-UL-P33 to R5-SS-P18

5 meters

R6 UL within Rack 6

R6-UL-P30 to R6-SS-P16

R6-UL-P32 to R6-SS-P18

3 meters

R6 UL to Rack 7

R6-UL-P5 to R7-SS-P16

5 meters

R6 LL to Rack 1

R6-LL-P5 to R1-SS-P12

R6-LL-P7 to R1-SS-P15

10 meters

R6 LL to Rack 2

R6-LL-P4 to R2-SS-P12

10 meters

R6 LL to Rack 3

R6-LL-P31 to R3-SS-P7

10 meters

R6 LL to Rack 4

R6-LL-P33 to R4-SS-P9

5 meters

R6 LL to Rack 5

R6-LL-P30 to R5-SS-P8

5 meters

R6 LL within Rack 6

R6-LL-P32 to R6-SS-P8

3 meters

R6 LL to Rack 7

R6-LL-P6 to R7-SS-P8

5 meters

The following table describes the cable connections for the seventh spine switch (R7-SS) when cabling seven full racks together:

Table 4-27 Leaf Switch Connections for the Seventh Rack in a Seven-Rack System

Leaf Switch Connection Cable Length

R7 UL to Rack 1

R7-UL-P5 to R1-SS-P9

10 meters

R7 UL to Rack 2

R7-UL-P7 to R2-SS-P7

10 meters

R7 UL to Rack 3

R7-UL-P4 to R3-SS-P9

10 meters

R7 UL to Rack 4

R7-UL-P6 to R4-SS-P8

10 meters

R7 UL to Rack 5

R7-UL-P31 to R5-SS-P9

5 meters

R7 UL to Rack 6

R7-UL-P33 to R6-SS-P9

5 meters

R7 UL within Rack 7

R7-UL-P30 to R7-SS-P9

R7-UL-P32 to R7-SS-P12

3 meters

R7 LL to Rack 1

R7-LL-P5 to R1-SS-P19

10 meters

R7 LL to Rack 2

R7-LL-P7 to R2-SS-P19

10 meters

R7 LL to Rack 3

R7-LL-P4 to R3-SS-P15

10 meters

R7 LL to Rack 4

R7-LL-P6 to R4-SS-P18

10 meters

R7 LL to Rack 5

R7-LL-P31 to R5-SS-P16

5 meters

R7 LL to Rack 6

R7-LL-P33 to R6-SS-P12

5 meters

R7 LL within Rack 7

R7-LL-P30 to R7-SS-P18

R7-LL-P32 to R7-SS-P19

3 meters

4.8 Eight-Rack Cabling forX8M Racks

This section provides the cabling tables to connect eight (8) Oracle Engineered System Racks together, all of which use InfiniBand Transport Layer systems based on a RoCE Network Layer.

Note:

The following conventions were used in the cabling notation for connecting multiple racks together.

  • The spine switch is in U1 (also referred to as SS) for all racks.

  • The leaf switches are referred to as Lower Leaf (LL) and Upper Leave (UL).

  • The leaf switches are located as follows:

    • At rack unit 21 (U21) and rack unit 23 (U23) in Oracle Engineered System X8M-8

    • At rack unit 20 (U20) and 22 (U22) in Oracle Engineered System X8M-2 racks or Storage Expansion Rack X8M-2

  • The cable lengths shown in the following lists assume that the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cable lengths. Cable lengths up to 100 meters are supported.

  • Only optical cables are supported for lengths greater than 5 meters

The following illustration shows the cable connections for the 8 spine switches when cabling eight racks together:



The following table describes the cable connections for the first spine switch (R1-SS) when cabling eight racks together:

Table 4-28 Leaf Switch Connections for the First Rack in a Eight-Rack System

Leaf Switch Connection Cable Length

R1 UL within Rack 1

R1-UL-P5 to R1-SS-P5

3 meters

R1 UL to Rack 2

R1-UL-P31 to R2-SS-P5

5 meters

R1 UL to Rack 3

R1-UL-P6 to R3-SS-P5

5 meters

R1 UL to Rack 4

R1-UL-P4 to R4-SS-P5

10 meters

R1 UL to Rack 5

R1-UL-P30 to R5-SS-P5

10 meters

R1 UL to Rack 6

R1-UL-P32 to R6-SS-P5

10 meters

R1 UL to Rack 7

R1-UL-P33 to R7-SS-P5

10 meters

R1 UL to Rack 8

R1-UL-P7 to R8-SS-P5

10 meters

R1 LL within Rack 1

R1-LL-P5 to R1-SS-P13

3 meters

R1 LL to Rack 2

R1-LL-P31 to R2-SS-P13

5 meters

R1 LL to Rack 3

R1-LL-P4 to R3-SS-P13

5 meters

R1 LL to Rack 4

R1-LL-P30 to R4-SS-P13

10 meters

R1 LL to Rack 5

R1-LL-P7 to R5-SS-P13

10 meters

R1 LL to Rack 6

R1-LL-P32 to R6-SS-P13

10 meters

R1 LL to Rack 7

R1-LL-P6 to R7-SS-P13

10 meters

R1 LL to Rack 8

R1-LL-P33 to R8-SS-P13

10 meters

The following table describes the cable connections for the second spine switch (R2-SS) when cabling eight full racks together:

Table 4-29 Leaf Switch Connections for the Second Rack in a Eight-Rack System

Leaf Switch Connection Cable Length

R2 UL to Rack 1

R2-UL-P5 to R1-SS-P6

5 meters

R2 UL within Rack 2

R2-UL-P31 to R2-SS-P6

3 meters

R2 UL to Rack 3

R2-UL-P6 to R3-SS-P6

5 meters

R2 UL to Rack 4

R2-UL-P4 to R4-SS-P6

5 meters

R2 UL to Rack 5

R2-UL-P7 to R5-SS-P6

10 meters

R2 UL to Rack 6

R2-UL-P33 to R6-SS-P6

10 meters

R2 UL to Rack 7

R2-UL-P32 to R7-SS-P6

10 meters

R2 UL to Rack 8

R2-UL-P30 to R8-SS-P6

10 meters

R2 LL to Rack 1

R2-LL-P5 to R1-SS-P14

5 meters

R2 LL within Rack 2

R2-LL-P31 to R2-SS-P14

3 meters

R2 LL to Rack 3

R2-LL-P6 to R3-SS-P14

5 meters

R2 LL to Rack 4

R2-LL-P4 to R4-SS-P14

5 meters

R2 LL to Rack 5

R2-LL-P33 to R5-SS-P14

10 meters

R2 LL to Rack 6

R2-LL-P7 to R6-SS-P14

10 meters

R2 LL to Rack 7

R2-LL-P32 to R7-SS-P14

10 meters

R2 LL to Rack 8

R2-LL-P30 to R8-SS-P14

10 meters

The following table describes the cable connections for the third spine switch (R3-SS) when cabling eight full racks together:

Table 4-30 Leaf Switch Connections for the Third Rack in a Eight-Rack System

Leaf Switch Connection Cable Length

R3 UL to Rack 1

R3-UL-P5 to R1-SS-P11

5 meters

R3 UL to Rack 2

R3-UL-P6 to R2-SS-P11

5 meters

R3 UL within Rack 3

R3-UL-P30 to R3-SS-P11

3 meters

R3 UL to Rack 4

R3-UL-P4 to R4-SS-P11

5 meters

R3 UL to Rack 5

R3-UL-P32 to R5-SS-P11

5 meters

R3 UL to Rack 6

R3-UL-P7 to R6-SS-P11

10 meters

R3 UL to Rack 7

R3-UL-P31 to R7-SS-P11

10 meters

R3 UL to Rack 8

R3-UL-P33 to R8-SS-P11

10 meters

R3 LL to Rack 1

R3-LL-P4 to R1-SS-P20

5 meters

R3 LL to Rack 2

R3-LL-P31 to R2-SS-P20

5 meters

R3 LL within Rack 3

R3-LL-P32 to R3-SS-P20

3 meters

R3 LL to Rack 4

R3-LL-P33 to R4-SS-P20

5 meters

R3 LL to Rack 5

R3-LL-P5 to R5-SS-P20

5 meters

R3 LL to Rack 6

R3-LL-P30 to R6-SS-P20

10 meters

R3 LL to Rack 7

R3-LL-P7 to R7-SS-P20

10 meters

R3 LL to Rack 8

R3-LL-P6 to R8-SS-P20

10 meters

The following table describes the cable connections for the fourth spine switch (R4-SS) when cabling eight full racks together:

Table 4-31 Leaf Switch Connections for the Fourth Rack in a Eight-Rack System

Leaf Switch Connection Cable Length

R4 UL to Rack 1

R4-UL-P7 to R1-SS-P10

10 meters

R4 UL to Rack 2

R4-UL-P6 to R2-SS-P10

5 meters

R4 UL to Rack 3

R4-UL-P33 to R3-SS-P10

5 meters

R4 UL within Rack 4

R4-UL-P32 to R4-SS-P10

3 meters

R4 UL to Rack 5

R4-UL-P5 to R5-SS-P10

5 meters

R4 UL to Rack 6

R4-UL-P30 to R6-SS-P10

5 meters

R4 UL to Rack 7

R4-UL-P31 to R7-SS-P10

10 meters

R4 UL to Rack 8

R4-UL-P4 to R8-SS-P10

10 meters

R4 LL to Rack 1

R4-LL-P5 to R1-SS-P17

10 meters

R4 LL to Rack 2

R4-LL-P4 to R2-SS-P17

5 meters

R4 LL to Rack 3

R4-LL-P31 to R3-SS-P17

5 meters

R4 LL within Rack 4

R4-LL-P30 to R4-SS-P17

3 meters

R4 LL to Rack 5

R4-LL-P7 to R5-SS-P17

5 meters

R4 LL to Rack 6

R4-LL-P6 to R6-SS-P17

5 meters

R4 LL to Rack 7

R4-LL-P32 to R7-SS-P17

10 meters

R4 LL to Rack 8

R4-LL-P33 to R8-SS-P17

10 meters

The following table describes the cable connections for the fifth spine switch (R5-SS) when cabling eight full racks together:

Table 4-32 Leaf Switch Connections for the Fifth Rack in a Eight-Rack System

Leaf Switch Connection Cable Length

R5 UL to Rack 1

R5-UL-P5 to R1-SS-P8

10 meters

R5 UL to Rack 2

R5-UL-P4 to R2-SS-P8

10 meters

R5 UL to Rack 3

R5-UL-P6 to R3-SS-P12

5 meters

R5 UL to Rack 4

R5-UL-P33 to R4-SS-P12

5 meters

R5 UL within Rack 5

R5-UL-P30 to R5-SS-P7

3 meters

R5 UL to Rack 6

R5-UL-P7 to R6-SS-P7

5 meters

R5 UL to Rack 7

R5-UL-P32 to R7-SS-P7

5 meters

R5 UL to Rack 8

R5-UL-P31 to R8-SS-P7

10 meters

R5 LL to Rack 1

R5-LL-P7 to R1-SS-P18

10 meters

R5 LL to Rack 2

R5-LL-P6 to R2-SS-P18

10 meters

R5 LL to Rack 3

R5-LL-P31 to R3-SS-P16

5 meters

R5 LL to Rack 4

R5-LL-P33 to R4-SS-P19

5 meters

R5 LL within Rack 5

R5-LL-P30 to R5-SS-P15

3 meters

R5 LL to Rack 6

R5-LL-P4 to R6-SS-P15

5 meters

R5 LL to Rack 7

R5-LL-P5 to R7-SS-P15

5 meters

R5 LL to Rack 8

R5-LL-P32 to R8-SS-P15

10 meters

The following table describes the cable connections for the sixth spine switch (R6-SS) when cabling eight full racks together:

Table 4-33 Leaf Switch Connections for the Sixth Rack in a Eight-Rack System

Leaf Switch Connection Cable Length

R6 UL to Rack 1

R6-UL-P7 to R1-SS-P16

10 meters

R6 UL to Rack 2

R6-UL-P4 to R2-SS-P16

10 meters

R6 UL to Rack 3

R6-UL-P6 to R3-SS-P19

10 meters

R6 UL to Rack 4

R6-UL-P31 to R4-SS-P15

5 meters

R6 UL to Rack 5

R6-UL-P33 to R5-SS-P18

5 meters

R6 UL within Rack 6

R6-UL-P30 to R6-SS-P16

3 meters

R6 UL to Rack 7

R6-UL-P5 to R7-SS-P16

5 meters

R6 UL to Rack 8

R6-UL-P32 to R8-SS-P16

5 meters

R6 LL to Rack 1

R6-LL-P5 to R1-SS-P12

10 meters

R6 LL to Rack 2

R6-UL-P4 to R2-SS-P12

10 meters

R6 LL to Rack 3

R6-LL-P31 to R3-SS-P7

10 meters

R6 LL to Rack 4

R6-LL-P33 to R4-SS-P9

5 meters

R6 LL to Rack 5

R6-LL-P30 to R5-SS-P8

5 meters

R6 LL within Rack 6

R6-LL-P32 to R6-SS-P8

3 meters

R6 LL to Rack 7

R6-LL-P6 to R7-SS-P8

5 meters

R6 LL to Rack 8

R6-LL-P7 to R8-SS-P8

5 meters

The following table describes the cable connections for the seventh spine switch (R7-SS) when cabling eight full racks together:

Table 4-34 Leaf Switch Connections for the Seventh Rack in a Eight-Rack System

Leaf Switch Connection Cable Length

R7 UL to Rack 1

R7-UL-P5 to R1-SS-P9

10 meters

R7 UL to Rack 2

R7-UL-P7 to R2-SS-P7

10 meters

R7 UL to Rack 3

R7-UL-P4 to R3-SS-P9

10 meters

R7 UL to Rack 4

R7-UL-P6 to R4-SS-P8

10 meters

R7 UL to Rack 5

R7-UL-P31 to R5-SS-P9

5 meters

R7 UL to Rack 6

R7-UL-P33 to R6-SS-P9

5 meters

R7 UL within Rack 7

R7-UL-P30 to R7-SS-P9

3 meters

R7 UL to Rack 8

R7-UL-P32 to R8-SS-P9

5 meters

R7 LL to Rack 1

R7-LL-P5 to R1-SS-P19

10 meters

R7 LL to Rack 2

R7-LL-P7 to R2-SS-P19

10 meters

R7 LL to Rack 3

R7-LL-P4 to R3-SS-P15

10 meters

R7 LL to Rack 4

R7-LL-P6 to R4-SS-P18

10 meters

R7 LL to Rack 5

R7-LL-P31 to R5-SS-P16

5 meters

R7 LL to Rack 6

R7-LL-P33 to R6-SS-P12

5 meters

R7 LL within Rack 7

R7-LL-P30 to R7-SS-P18

3 meters

R7 LL to Rack 8

R7-LL-P32 to R8-SS-P12

5 meters

The following table describes the cable connections for the eighth spine switch (R8-SS) when cabling eight full racks together:

Table 4-35 Leaf Switch Connections for the Eighth Rack in a Eight-Rack System

Leaf Switch Connection Cable Length

R8 UL to Rack 1

R8-UL-P5 to R1-SS-P15

10 meters

R8 UL to Rack 2

R8-UL-P7 to R2-SS-P15

10 meters

R8 UL to Rack 3

R8-UL-P4 to R3-SS-P18

10 meters

R8 UL to Rack 4

R8-UL-P6 to R4-SS-P16

10 meters

R8 UL to Rack 5

R8-UL-P31 to R5-SS-P19

10 meters

R8 UL to Rack 6

R8-UL-P33 to R6-SS-P19

5 meters

R8 UL to Rack 7

R8-UL-P30 to R7-SS-P19

5 meters

R8 UL within Rack 8

R8-UL-P32 to R8-SS-P19

3 meters

R8 LL to Rack 1

R8-LL-P5 to R1-SS-P7

10 meters

R8 LL to Rack 2

R8-LL-P7 to R2-SS-P9

10 meters

R8 LL to Rack 3

R8-LL-P4 to R3-SS-P12

10 meters

R8 LL to Rack 4

R8-LL-P6 to R4-SS-P7

10 meters

R8 LL to Rack 5

R8-LL-P31 to R5-SS-P12

10 meters

R8 LL to Rack 6

R8-LL-P33 to R6-SS-P12

5 meters

R8 LL to Rack 7

R8-LL-P30 to R7-SS-P12

5 meters

R8 LL within Rack 8

R8-LL-P32 to R8-SS-P18

3 meters