6 Multi-Rack Cabling Tables for Oracle Exadata Rack Models with InfiniBand Network Fabric (X2 to X8)
This section contains multi-rack cabling tables for Oracle Exadata Rack models that use InfiniBand Network Fabric. This includes Oracle Exadata Rack models from X2 to X8.
- Understanding Multi-Rack Cabling for Racks with InfiniBand Network Fabric
Up to eight racks can be cabled together without external RDMA Network Fabric switches. - Two-Rack Cabling with InfiniBand Network Fabric
Review this information before cabling two racks together with InfiniBand Network Fabric. - Three-Rack Cabling with InfiniBand Network Fabric
- Four-Rack Cabling with InfiniBand Network Fabric
- Five-Rack Cabling with InfiniBand Network Fabric
- Six-Rack Cabling with InfiniBand Network Fabric
- Seven-Rack Cabling with InfiniBand Network Fabric
- Eight-Rack Cabling with InfiniBand Network Fabric
6.1 Understanding Multi-Rack Cabling for Racks with InfiniBand Network Fabric
Up to eight racks can be cabled together without external RDMA Network Fabric switches.
Three RDMA Network Fabric switches are needed when cabling racks together. If any rack does not include a third switch, then order the expansion switch kit. These switches attach to standard RDMA Network Fabric connectors at the end of the cables. The procedures in this section assume the racks are adjacent to each other. If they are not, then longer cables may be required for the connections.
Note:
Only for InfiniBand Network Fabric (X8 and earlier).-
For Eighth or Quarter Racks, which are the smallest Elastic Configurations, follow the instructions in "Cabling Oracle Exadata Quarter Racks and Oracle Exadata Eighth Racks with InfiniBand Network Fabric" for direct connection without spine switch.
-
For other racks (Half Rack, Full Rack, Elastic Configurations larger than Eighth or Quarter Rack) install a spine switch and follow the standard N-rack cabling for two or more interconnected racks, for example "Two-Rack Cabling with InfiniBand Network Fabric."
The switch at rack unit 1 (U1) is referred to as the spine switch.
The following switches are referred to as leaf switches:
-
The switches at rack unit 20 (U20) and rack unit 24 (U24) in Oracle Exadata X2-2 racks and Oracle Exadata Storage Expansion Racks
-
The switches at rack unit 21 (U21) and rack unit 23 (U23) in Oracle Exadata X2-8 and later.
-
The switches at rack unit 20 (U20) and 22 (U22) in Oracle Exadata X3-2 and later racks or Storage Expansion Rack X3-2 and later.
Note:
Oracle Exadata Database Machine X4-2 and later racks or Oracle Exadata Database Machine X3-8 Full Racks with Exadata Storage Server X4-2L Servers do not include spare cables or a third Sun Datacenter InfiniBand Switch 36 switch. To extend Oracle Exadata Database Machine X4-2 and later racks or Oracle Exadata Database Machine X3-8 Full Racks with Exadata Storage Server X4-2L Servers, you must order cables and a Sun Datacenter InfiniBand Switch 36 switch.
In a single rack, the two leaf switches are interconnected using seven connections. In addition, each leaf switch has one connection to the spine switch. The leaf switches connect to the spine switch as shown in the following graphic:
Figure 6-1 Connections Between Spine Switch and Leaf Switches in a Single Rack
Description of "Figure 6-1 Connections Between Spine Switch and Leaf Switches in a Single Rack"
The Oracle Database servers and Exadata Storage Servers connect to the leaf switches as shown in the following graphic:
Figure 6-2 Connections Between Database Servers and Storage Servers and Leaf Switches
Description of "Figure 6-2 Connections Between Database Servers and Storage Servers and Leaf Switches"
When connecting up to eight racks together, remove the seven existing inter-switch connections between each leaf switch, as well as the two connections between the leaf switches and the spine switch. From each leaf switch, distribute eight connections over the spine switches in all racks. In multi-rack environments, the leaf switches inside a rack are no longer directly interconnected, as shown in the following graphic:
Figure 6-3 Connections Between Spine Switches and Leaf Switches Across Two Racks
Description of "Figure 6-3 Connections Between Spine Switches and Leaf Switches Across Two Racks"
As shown in the preceding graphic, each leaf switch in rack 1 connects to the following switches:
-
Four connections to its internal spine switch
-
Four connections to the spine switch in rack 2
The spine switch in rack 1 connects to the following switches:
-
Eight connections to both internal leaf switches
-
Eight connections to both leaf switches in rack 2
As the number of racks increases from two to eight, the pattern continues as shown in the following graphic:
Figure 6-4 Connections Between Spine Switches and Leaf Switches for up to 8 Racks
Description of "Figure 6-4 Connections Between Spine Switches and Leaf Switches for up to 8 Racks"
As shown in the preceding graphic, each leaf switch has eight inter-switch connections distributed over all spine switches. Each spine switch has 16 inter-switch connections distributed over all leaf switches. The leaf switches are not directly interconnected with other leaf switches, and the spine switches are not directly interconnected with the other spine switches.
Note:
Cable lengths are specified for racks 1 through 8.
6.1.1 Preparing for Multi-Rack Cabling with InfiniBand Network Fabric
Racks can be added together to increase system capacity and performance. When cabling racks together, note the following:
-
The cable lengths shown in this document assume the racks are adjacent to each other. If the racks are not adjacent, or there are obstacles in the raised floor, or if you use overhead cabling, then longer cables may be required. For optical cables, the maximum supported cable length is 100 meters. For copper cables, the maximum supported cable length is 5 meters.
-
Oracle recommends that the names for the servers include the rack unit number. This helps identify the server during diagnostics.
-
When completing Oracle Exadata Deployment Assistant for the additional rack, you are prompted for SCAN addresses. However, these SCAN addresses are not used because the SCAN address from the original rack are used. Manually remove the new SCAN addresses from the generated installation files.
-
The software owner account names and group names, as well as their identifiers, must match the names and identifiers of the original rack.
-
If the additional grid disks are used with existing disk groups, then ensure the grid disk sizes for the new rack are the same as the original rack.
-
If the InfiniBand network consists of four or more racks cabled together, then disable the Subnet Manager on the leaf switches.
-
Verify the Master Subnet Manager is located on the spine switch.
-
Oracle Exadata Database Machine Quarter Racks can be extended as follows:
-
Connect two Oracle Exadata Database Machine Quarter Racks together. At least four of the six ports reserved for external connectivity are open on each leaf switch. The six ports are 5B, 6A, 6B, 7A, 7B, and 12A in each leaf switch. Maintain the existing seven inter-switch links between the leaf switches within each rack. Connect the leaf switches between the racks with two links each, using the ports reserved for external connectivity.
-
Connect one Oracle Exadata Database Machine Quarter Rack with one Oracle Exadata Database Machine Half Rack or one Oracle Exadata Database Machine Full Rack. At least four ports reserved for external connectivity are open on each leaf switch. The spine switch in the Oracle Exadata Database Machine Half Rack or Oracle Exadata Database Machine Full Rack remains as the spine switch. Maintain the existing seven inter-switch links between the leaf switches within each rack. Connect the leaf switches between the racks with two links each, using the ports reserved for external connectivity.
-
Connect one Oracle Exadata Database Machine Quarter Rack with two or more Oracle Exadata Database Machine Half Racks or Oracle Exadata Database Machine Full Racks. The racks are interconnected using a fat-tree topology. Connect each leaf switch in the quarter rack to the spine switch of each half rack or full rack using two links each. If there are more than four racks, then use one link instead of two. The seven inter-switch links between the leaf switches in the quarter rack are removed.
Note:
To connect more than one quarter rack to additional racks, it is necessary to purchase Sun Datacenter InfiniBand Switch 36 switches for the quarter racks.
-
-
If you are extending Oracle Exadata Database Machine X4-2 or later, or Oracle Exadata Database Machine X3-8 Full Rack, or Oracle Exadata Database Machine X2-2 (with X4170 and X4275 servers) half rack, then order the expansion kit that includes a Sun Datacenter InfiniBand Switch 36 switch.
Perform the following tasks before cabling racks together:
-
Determine the number of racks that will be cabled together.
-
Count the spare cables from the kit, and existing inter-switch cables.
Note:
Oracle Exadata Database Machine X4-2 and later racks or Oracle Exadata Database Machine X3-8 Full Racks with Exadata Storage Server X4-2L Servers do not include spare cables or a third Sun Datacenter InfiniBand Switch 36 switch. To extend Oracle Exadata Database Machine X4-2 and later racks or Oracle Exadata Database Machine X3-8 Full Racks with Exadata Storage Server X4-2L Servers, you must order cables and a Sun Datacenter InfiniBand Switch 36 switch.
For Oracle Exadata Racks earlier than Oracle Exadata Database Machine X4-2, no additional InfiniBand cables need to be purchased when connecting up to three Oracle Exadata Database Machine Full Racks. The following table lists the spare cables for the switch:
Cable Spare Inter-switch Total InfiniBand 2 meter cable
0
7
7
InfiniBand 3 meter cable
6
2
8
InfiniBand 5 meter cable
10
0
10
When connecting four or more racks, it is necessary to purchase additional 10 meter optical QSFP-to-QSFP QDR InfiniBand cables. Two QSFP SR transceivers are needed for each optical cable. The following table lists the number of racks, and cables needed to interconnect the racks.
Number of Racks 3 Meter Cables Needed 5 Meter Cables Needed 10 Meter Cables Needed 2
16
16
0
3
18
30
0
4
16
40
8
5
20
42
18
6
24
46
26
7
28
44
40
8
16
52
60
-
Gather the following tools:
-
Pen
-
Note pad
-
Flashlight
-
Labels for cables and rack
-
Cable ties (Oracle recommends hook and loop straps)
-
-
Determine a naming method for the rack prefixes. For example, if the original rack has the prefix
dbm01, then use the prefixdbm02for the second rack, the prefixdbm03for the third rack, and so on. -
Verify the racks have unique host names and IP addresses. All servers interconnected in the racks must have unique names and IP addresses.
Server names and IP addresses conventions may differ in the following cases:
-
Initial installation of all Oracle Exadata Database Machine Full Racks: System address assignments and host names should be complete.
-
New Oracle Exadata Database Machine Full Racks are added to an existing cluster: The new rack configuration should require unique host names and IP addresses for the new Oracle Exadata Database Machines. The IP addresses on the same subnet cannot conflict with the existing systems.
-
Two existing Oracle Exadata Database Machine Full Racks are clustered together: You can assign host names and IP addresses only if Oracle Exadata Database Machines are already assigned unique host names and IP addresses, or the entire cluster must be reconfigured. The machines must be on the same subnet and not have conflicting IP addresses.
-
-
Ensure the IP addresses for the new servers are in the same subnet, and do not overlap with the currently-installed servers.
-
Ensure the firmware on the original switches are at the same level as the new switches using the
nm2versioncommand. If the firmware is not at the same level, then apply a firmware patch.
6.1.2 Cabling Oracle Exadata Quarter Racks and Oracle Exadata Eighth Racks with InfiniBand Network Fabric
Oracle Exadata Quarter Racks and Oracle Exadata Eighth Racks with InfiniBand Network Fabric can be cabled as follows:
-
Oracle Exadata Quarter Rack to Oracle Exadata Quarter Rack
-
Oracle Exadata Quarter Rack to Oracle Exadata Half Rack, or multiple Oracle Exadata Half Racks or Oracle Exadata Full Racks
-
Oracle Exadata Quarter Rack to Oracle Exadata Full Rack, or multiple Oracle Exadata Full Racks or Oracle Exadata Half Racks
-
Oracle Exadata Eighth Rack to Oracle Exadata Eighth Rack
-
Oracle Exadata Eighth Rack to Oracle Exadata Half Rack, or multiple Oracle Exadata Half Racks or Oracle Exadata Full Racks
-
Oracle Exadata Eighth Rack to Oracle Exadata Full Rack, or multiple Oracle Exadata Full Racks or Oracle Exadata Half Racks
Note:
-
To cable additional Oracle Exadata Quarter Racks or Oracle Exadata Eighth Racks, it is necessary to purchase additional Sun Datacenter InfiniBand Switch 36 switches.
-
It is assumed that four of the six ports dedicated for external connectivity in Oracle Exadata Half Rack or Oracle Exadata Full Rack are unused and available for switch interconnections with Oracle Exadata Quarter Rack or Oracle Exadata Eighth Rack. The six ports are 5B, 6A, 6B, 7A, 7B, and 12A in each leaf switch.
The following graphic shows the cable connections for two Oracle Exadata Quarter Racks. The leaf switches within each rack maintain their existing seven connections. The leaf switches interconnect between the racks with two links each using the ports reserved for external connectivity.
Figure 6-5 Leaf Switch Connections Between 2 Quarter Racks
Description of "Figure 6-5 Leaf Switch Connections Between 2 Quarter Racks"
The following graphic shows the cable connections from Oracle Exadata Quarter Rack to Oracle Exadata Half Rack or Oracle Exadata Full Rack. The leaf switches within each rack maintain their existing seven connections. The leaf switches interconnect between the racks with two links each using the ports reserved for external connectivity.
Figure 6-6 Leaf and Spine Switch Connections Between a Quarter Rack and a Half or Full Rack
Description of "Figure 6-6 Leaf and Spine Switch Connections Between a Quarter Rack and a Half or Full Rack"
The following graphic shows the cable connections from Oracle Exadata Quarter Rack to two or more racks. The racks that connect to Oracle Exadata Quarter Rack must be all Oracle Exadata Half Racks or Oracle Exadata Full Racks, interconnected using a fat-tree topology. Each leaf switch in Oracle Exadata Quarter Rack connects to the spine switches in the other half racks or full racks with two links each. If there are more than four racks, then use one link instead of two. The seven inter-switch links between the leaf switches in the quarter rack are removed.
Figure 6-7 Leaf and Spine Switch Connections for a Quarter Rack Connected to One or More Half or Full Racks
Description of "Figure 6-7 Leaf and Spine Switch Connections for a Quarter Rack Connected to One or More Half or Full Racks"
6.2 Two-Rack Cabling with InfiniBand Network Fabric
Review this information before cabling two racks together with InfiniBand Network Fabric.
-
The spine switch is in U1 (also referred to as IB1) for all racks.
-
In the following lists, the leaf switches are referred to as IB2 and IB3. Use the rack unit that is appropriate for your system.
-
In Oracle Exadata Database Machine X2-2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers, the leaf switches are in U20 and U24, also referred to as IB2 and IB3, respectively.
-
In Oracle Exadata Database Machine 8-socket (X8-8, X7-8, X6-8, X5-8, X4-8, X3-8 and X2-8) Full Rack systems, the leaf switches are in U21 and U23, also referred to as IB2 and IB3, respectively.
-
In Oracle Exadata Database Machine X4-2 and later racks, Oracle Exadata Database Machine X3-2 Full Racks and Oracle Exadata Storage Expansion Rack X3-2 racks, the leaf switches are in U20 and U22, also referred to as IB2 and IB3, respectively.
-
The cable lengths shown in the following lists assume that the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cables lengths. Cables lengths up to 100 meters are supported.
The following table shows the cable connections for the first spine switch (R1-U1) when cabling two full racks together.
Table 6-1 Leaf Switch Connections for the First Rack in a Two-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
| R1 IB3 within Rack 1 |
R1-IB3-P8A to R1-U1-P3A |
3 meters |
| R1 IB3 to Rack 2 |
R1-IB3-P10A to R2-U1-P7A |
5 meters |
| R1 IB2 within Rack 1 |
R1-IB2-P8A to R1-U1-P3B |
3 meters |
| R1 IB2 to Rack 2 |
R1-IB2-P10A to R2-U1-P7B |
5 meters |
The following table shows the cable connections for the second spine switch (R2-U1) when cabling two full racks together.
Table 6-2 Leaf Switch Connections for the Second Rack in a Two-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
| R2 IB3 within Rack 2 |
R2-IB3-P8A to R2-U1-P3A |
3 meters |
| R2 IB3 to Rack 1 |
R2-IB3-P10A to R1-U1-P7A |
5 meters |
| R2 IB2 within Rack 2 |
R2-IB2-P8A to R2-U1-P3B |
3 meters |
| R2 IB2 to Rack 1 |
R2-IB2-P10A to R1-U1-P7B |
5 meters |
6.3 Three-Rack Cabling with InfiniBand Network Fabric
Note the following when cabling three racks together with InfiniBand Network Fabric:
-
The spine switch is in U1 (also referred to as IB1) for all racks.
-
In the following tables, the leaf switches are referred to as IB2 and IB3. Use the rack unit that is appropriate for your system.
-
In Oracle Exadata Database Machine X2-2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers, the leaf switches are in U20 and U24, also referred to as IB2 and IB3, respectively.
-
In Oracle Exadata Database Machine X2-8 and later, the leaf switches are in U21 and U23, also referred to as IB2 and IB3, respectively.
-
In Oracle Exadata Database Machine X4-2 and later racks, or Oracle Exadata Database Machine X3-2 Full Racks or Oracle Exadata Storage Expansion Rack X3-2 racks, the leaf switches are in U20 and U22, also referred to as IB2 and IB3, respectively.
-
The cable lengths shown in the tables assume the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cables lengths. Up to 100 meters is supported.
-
Only optical cables are supported for lengths greater than 5 meters.
Table 6-3 shows the cable connections for the first spine switch (R1-U1) when cabling three racks together:
Table 6-3 Leaf Switch Connections for the First Rack in a Three-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R1 IB3 within Rack 1 |
R1-IB3-P8A to R1-U1-P3A R1-IB3-P8B to R1-U1-P4A R1-IB3-P9A to R1-U1-P5A |
3 meters |
|
R1 IB3 to Rack 2 |
R1-IB3-P9B to R2-U1-P6A R1-IB3-P10A to R2-U1-P7A R1-IB3-P10B to R2-U1-P8A |
5 meters |
|
R1 IB3 to Rack 3 |
R1-IB3-P11A to R3-U1-P9A R1-IB3-P11B to R3-U1-P10A |
5 meters |
|
R1 IB2 within Rack 1 |
R1-IB2-P8A to R1-U1-P3B R1-IB2-P8B to R1-U1-P4B R1-IB2-P9A to R1-U1-P5B |
3 meters |
|
R1 IB2 to Rack 2 |
R1-IB2-P9B to R2-U1-P6B R1-IB2-P10A to R2-U1-P7B R1-IB2-P10B to R2-U1-P8B |
5 meters |
|
R1 IB2 to Rack 3 |
R1-IB2-P11A to R3-U1-P9B R1-IB2-P11B to R3-U1-P10B |
5 meters |
Table 6-4 shows the cable connections for the second spine switch (R2-U1) when cabling three racks together:
Table 6-4 Leaf Switch Connections for the Second Rack in a Three-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R2 IB3 within Rack 2 |
R2-IB3-P8A to R2-U1-P3A R2-IB3-P8B to R2-U1-P4A R2-IB3-P9A to R2-U1-P5A |
3 meters |
|
R2 IB3 to Rack 1 |
R2-IB3-P11A to R1-U1-P9A R2-IB3-P11B to R1-U1-P10A |
5 meters |
|
R2 IB3 to Rack 3 |
R2-IB3-P9B to R3-U1-P6A R2-IB3-P10A to R3-U1-P7A R2-IB3-P10B to R3-U1-P8A |
5 meters |
|
R2 IB2 within Rack 2 |
R2-IB2-P8A to R2-U1-P3B R2-IB2-P8B to R2-U1-P4B R2-IB2-P9A to R2-U1-P5B |
3 meters |
|
R2 IB2 to Rack 1 |
R2-IB2-P11A to R1-U1-P9B R2-IB2-P11B to R1-U1-P10B |
5 meters |
|
R2 IB2 to Rack 3 |
R2-IB2-P9B to R3-U1-P6B R2-IB2-P10A to R3-U1-P7B R2-IB2-P10B to R3-U1-P8B |
5 meters |
Table 6-5 shows the cable connections for the third spine switch (R3-U1) when cabling three full racks together:
Table 6-5 Leaf Switch Connections for the Third Rack in a Three-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R3 IB3 within Rack 3 |
R3-IB3-P8A to R3-U1-P3A R3-IB3-P8B to R3-U1-P4A R3-IB3-P9A to R3-U1-P5A |
3 meters |
|
R3 IB3 to Rack 1 |
R3-IB3-P9B to R1-U1-P6A R3-IB3-P10A to R1-U1-P7A R3-IB3-P10B to R1-U1-P8A |
5 meters |
|
R3 IB3 to Rack 2 |
R3-IB3-P11A to R2-U1-P9A R3-IB3-P11B to R2-U1-P10A |
5 meters |
|
R3 IB2 within Rack 3 |
R3-IB2-P8A to R3-U1-P3B R3-IB2-P8B to R3-U1-P4B R3-IB2-P9A to R3-U1-P5B |
3 meters |
|
R3 IB2 to Rack 1 |
R3-IB2-P9B to R1-U1-P6B R3-IB2-P10A to R1-U1-P7B R3-IB2-P10B to R1-U1-P8B |
5 meters |
|
R3 IB2 to Rack 2 |
R3-IB2-P11A to R2-U1-P9B R3-IB2-P11B to R2-U1-P10B |
5 meters |
6.4 Four-Rack Cabling with InfiniBand Network Fabric
Note the following when cabling four racks together with InfiniBand Network Fabric:
-
The spine switch is in U1 (also referred to as IB1) for all racks.
-
In the following tables, the leaf switches are referred to as IB2 and IB3. Use the rack unit that is appropriate for your system.
-
In Oracle Exadata Database Machine X2-2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers, the leaf switches are in U20 and U24, also referred to as IB2 and IB3, respectively.
-
In Oracle Exadata Database Machine X2-8 and later, the leaf switches are in U21 and U23, also referred to as IB2 and IB3, respectively.
-
In Oracle Exadata Database Machine X4-2 and later racks, or Oracle Exadata Database Machine X3-2 Full Racks or Oracle Exadata Storage Expansion Rack X3-2 racks, the leaf switches are in U20 and U22, also referred to as IB2 and IB3, respectively.
-
The cable lengths shown in the tables assume the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cables lengths. Up to 100 meters is supported.
-
Only optical cables are supported for lengths greater than 5 meters.
Table 6-6 shows the cable connections for the first spine switch (R1-U1) when cabling four racks together:
Table 6-6 Leaf Switch Connections for the First Rack in a Four-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R1 IB3 within Rack 1 |
R1-IB3-P8A to R1-U1-P3A R1-IB3-P8B to R1-U1-P4A |
3 meters |
|
R1 IB3 to Rack 2 |
R1-IB3-P9A to R2-U1-P5A R1-IB3-P9B to R2-U1-P6A |
5 meters |
|
R1 IB3 to Rack 3 |
R1-IB3-P10A to R3-U1-P7A R1-IB3-P10B to R3-U1-P8A |
5 meters |
|
R1 IB3 to Rack 4 |
R1-IB3-P11A to R4-U1-P9A R1-IB3-P11B to R4-U1-P10A |
10 meters |
|
R1 IB2 within Rack 1 |
R1-IB2-P8A to R1-U1-P3B R1-IB2-P8B to R1-U1-P4B |
3 meters |
|
R1 IB2 to Rack 2 |
R1-IB2-P9A to R2-U1-P5B R1-IB2-P9B to R2-U1-P6B |
5 meters |
|
R1 IB2 to Rack 3 |
R1-IB2-P10A to R3-U1-P7B R1-IB2-P10B to R3-U1-P8B |
5 meters |
|
R1 IB2 to Rack 4 |
R1-IB2-P11A to R4-U1-P9B R1-IB2-P11B to R4-U1-P10B |
10 meters |
Table 6-7 shows the cable connections for the second spine switch (R2-U1) when cabling four full racks together:
Table 6-7 Leaf Switch Connections for the Second Rack in a Four-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R2 IB3 within Rack 2 |
R2-IB3-P8A to R2-U1-P3A R2-IB3-P8B to R2-U1-P4A |
3 meters |
|
R2 IB3 to Rack 1 |
R2-IB3-P11A to R1-U1-P9A R2-IB3-P11B to R1-U1-P10A |
5 meters |
|
R2 IB3 to Rack 3 |
R2-IB3-P9A to R3-U1-P5A R2-IB3-P9B to R3-U1-P6A |
5 meters |
|
R2 IB3 to Rack 4 |
R2-IB3-P10A to R4-U1-P7A R2-IB3-P10B to R4-U1-P8A |
5 meters |
|
R2 IB2 within Rack 2 |
R2-IB2-P8A to R2-U1-P3B R2-IB2-P8B to R2-U1-P4B |
3 meters |
|
R2 IB2 to Rack 1 |
R2-IB2-P11A to R1-U1-P9B R2-IB2-P11B to R1-U1-P10B |
5 meters |
|
R2 IB2 to Rack 3 |
R2-IB2-P9A to R3-U1-P5B R2-IB2-P9B to R3-U1-P6B |
5 meters |
|
R2 IB2 to Rack 4 |
R2-IB2-P10A to R4-U1-P7B R2-IB2-P10B to R4-U1-P8B |
5 meters |
Table 6-8 shows the cable connections for the third spine switch (R3-U1) when cabling four full racks together:
Table 6-8 Leaf Switch Connections for the Third Rack in a Four-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R3 IB3 within Rack 3 |
R3-IB3-P8A to R3-U1-P3A R3-IB3-P8B to R3-U1-P4A |
3 meters |
|
R3 IB3 to Rack 1 |
R3-IB3-P10A to R1-U1-P7A R3-IB3-P10B to R1-U1-P8A |
5 meters |
|
R3 IB3 to Rack 2 |
R3-IB3-P11A to R2-U1-P9A R3-IB3-P11B to R2-U1-P10A |
5 meters |
|
R3 IB3 to Rack 4 |
R3-IB3-P9A to R4-U1-P5A R3-IB3-P9B to R4-U1-P6A |
5 meters |
|
R3 IB2 within Rack 3 |
R3-IB2-P8A to R3-U1-P3B R3-IB2-P8B to R3-U1-P4B |
3 meters |
|
R3 IB2 to Rack 1 |
R3-IB2-P10A to R1-U1-P7B R3-IB2-P10B to R1-U1-P8B |
5 meters |
|
R3 IB2 to Rack 2 |
R3-IB2-P11A to R2-U1-P9B R3-IB2-P11B to R2-U1-P10B |
5 meters |
|
R3 IB2 to Rack 4 |
R3-IB2-P9A to R4-U1-P5B R3-IB2-P9B to R4-U1-P6B |
5 meters |
Table 6-9 shows the cable connections for the fourth spine switch (R4-U1) when cabling four full racks together:
Table 6-9 Leaf Switch Connections for the Fourth Rack in a Four-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R4 IB3 within Rack 4 |
R4-IB3-P8A to R4-U1-P3A R4-IB3-P8B to R4-U1-P4A |
3 meters |
|
R4 IB3 to Rack 1 |
R4-IB3-P9A to R1-U1-P5A R4-IB3-P9B to R1-U1-P6A |
10 meters |
|
R4 IB3 to Rack 2 |
R4-IB3-P10A to R2-U1-P7A R4-IB3-P10B to R2-U1-P8A |
5 meters |
|
R4 IB3 to Rack 3 |
R4-IB3-P11A to R3-U1-P9A R4-IB3-P11B to R3-U1-P10A |
5 meters |
|
R4 IB2 within Rack 4 |
R4-IB2-P8A to R4-U1-P3B R4-IB2-P8B to R4-U1-P4B |
3 meters |
|
R4 IB2 to Rack 1 |
R4-IB2-P9A to R1-U1-P5B R4-IB2-P9B to R1-U1-P6B |
10 meters |
|
R4 IB2 to Rack 2 |
R4-IB2-P10A to R2-U1-P7B R4-IB2-P10B to R2-U1-P8B |
5 meters |
|
R4 IB2 to Rack 3 |
R4-IB2-P11A to R3-U1-P9B R4-IB2-P11B to R3-U1-P10B |
5 meters |
6.5 Five-Rack Cabling with InfiniBand Network Fabric
Note the following when cabling five racks together with InfiniBand Network Fabric:
-
The spine switch is in U1 (also referred to as IB1) for all racks.
-
In the following tables, the leaf switches are referred to as IB2 and IB3. Use the rack unit that is appropriate for your system.
-
In Oracle Exadata Database Machine X2-2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers, the leaf switches are in U20 and U24, also referred to as IB2 and IB3, respectively.
-
In Oracle Exadata Database Machine X2-8 and later racks, the leaf switches are in U21 and U23, also referred to as IB2 and IB3, respectively.
-
In Oracle Exadata Database Machine X4-2 and later racks, or Oracle Exadata Database Machine X3-2 Full Racks and Oracle Exadata Storage Expansion Rack X3-2 racks, the leaf switches are in U20 and U22, also referred to as IB2 and IB3, respectively.
-
The cable lengths shown in the tables assume the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cables lengths. Up to 100 meters is supported.
-
Only optical cables are supported for lengths greater than 5 meters.
Table 6-10 shows the cable connections for the first spine switch (R1-U1) when cabling five racks together:
Table 6-10 Leaf Switch Connections for the First Rack in a Five-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R1 IB3 within Rack 1 |
R1-IB3-P8A to R1-U1-P3A R1-IB3-P8B to R1-U1-P4A |
3 meters |
|
R1 IB3 to Rack 2 |
R1-IB3-P9A to R2-U1-P5A R1-IB3-P9B to R2-U1-P6A |
5 meters |
|
R1 IB3 to Rack 3 |
R1-IB3-P10A to R3-U1-P7A R1-IB3-P10B to R3-U1-P8A |
5 meters |
|
R1 IB3 to Rack 4 |
R1-IB3-P11A to R4-U1-P9A |
10 meters |
|
R1 IB3 to Rack 5 |
R1-IB3-P11B to R5-U1-P10A |
10 meters |
|
R1 IB2 within Rack 1 |
R1-IB2-P8A to R1-U1-P3B R1-IB2-P8B to R1-U1-P4B |
3 meters |
|
R1 IB2 to Rack 2 |
R1-IB2-P9A to R2-U1-P5B R1-IB2-P9B to R2-U1-P6B |
5 meters |
|
R1 IB2 to Rack 3 |
R1-IB2-P10A to R3-U1-P7B R1-IB2-P10B to R3-U1-P8B |
5 meters |
|
R1 IB2 to Rack 4 |
R1-IB2-P11A to R4-U1-P9B |
10 meters |
|
R1 IB2 to Rack 5 |
R1-IB2-P11B to R5-U1-P10B |
10 meters |
Table 6-11 shows the cable connections for the second spine switch (R2-U1) when cabling five full racks together:
Table 6-11 Leaf Switch Connections for the Second Rack in a Five-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R2 IB3 within Rack 2 |
R2-IB3-P8A to R2-U1-P3A R2-IB3-P8B to R2-U1-P4A |
3 meters |
|
R2 IB3 to Rack 1 |
R2-IB3-P11B to R1-U1-P10A |
5 meters |
|
R2 IB3 to Rack 3 |
R2-IB3-P9A to R3-U1-P5A R2-IB3-P9B to R3-U1-P6A |
5 meters |
|
R2 IB3 to Rack 4 |
R2-IB3-P10A to R4-U1-P7A R2-IB3-P10B to R4-U1-P8A |
5 meters |
|
R2 IB3 to Rack 5 |
R2-IB3-P11A to R5-U1-P9A |
10 meters |
|
R2 IB2 within Rack 2 |
R2-IB2-P8A to R2-U1-P3B R2-IB2-P8B to R2-U1-P4B |
3 meters |
|
R2 IB2 to Rack 1 |
R2-IB2-P11B to R1-U1-P10B |
5 meters |
|
R2 IB2 to Rack 3 |
R2-IB2-P9A to R3-U1-P5B R2-IB2-P9B to R3-U1-P6B |
5 meters |
|
R2 IB2 to Rack 4 |
R2-IB2-P10A to R4-U1-P7B R2-IB2-P10B to R4-U1-P8B |
5 meters |
|
R2 IB2 to Rack 5 |
R2-IB2-P11A to R5-U1-P9B |
10 meters |
Table 6-12 shows the cable connections for the third spine switch (R3-U1) when cabling five full racks together:
Table 6-12 Leaf Switch Connections for the Third Rack in a Five-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R3 IB3 within Rack 3 |
R3-IB3-P8A to R3-U1-P3A R3-IB3-P8B to R3-U1-P4A |
3 meters |
|
R3 IB3 to Rack 1 |
R3-IB3-P11A to R1-U1-P9A |
5 meters |
|
R3 IB3 to Rack 2 |
R3-IB3-P11B to R2-U1-P10A |
5 meters |
|
R3 IB3 to Rack 4 |
R3-IB3-P9A to R4-U1-P5A R3-IB3-P9B to R4-U1-P6A |
5 meters |
|
R3 IB3 to Rack 5 |
R3-IB3-P10A to R5-U1-P7A R3-IB3-P10B to R5-U1-P8A |
5 meters |
|
R3 IB2 within Rack 3 |
R3-IB2-P8A to R3-U1-P3B R3-IB2-P8B to R3-U1-P4B |
3 meters |
|
R3 IB2 to Rack 1 |
R3-IB2-P11A to R1-U1-P9B |
5 meters |
|
R3 IB2 to Rack 2 |
R3-IB2-P11B to R2-U1-P10B |
5 meters |
|
R3 IB2 to Rack 4 |
R3-IB2-P9A to R4-U1-P5B R3-IB2-P9B to R4-U1-P6B |
5 meters |
|
R3 IB2 to Rack 5 |
R3-IB2-P10A to R5-U1-P7B R3-IB2-P10B to R5-U1-P8B |
5 meters |
Table 6-13 shows the cable connections for the fourth spine switch (R4-U1) when cabling five full racks together:
Table 6-13 Leaf Switch Connections for the Fourth Rack in a Five-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R4 IB3 within Rack 4 |
R4-IB3-P8A to R4-U1-P3A R4-IB3-P8B to R4-U1-P4A |
3 meters |
|
R4 IB3 to Rack 1 |
R4-IB3-P10A to R1-U1-P7A R4-IB3-P10B to R1-U1-P8A |
10 meters |
|
R4 IB3 to Rack 2 |
R4-IB3-P11A to R2-U1-P9A |
5 meters |
|
R4 IB3 to Rack 3 |
R4-IB3-P11B to R3-U1-P10A |
5 meters |
|
R4 IB3 to Rack 5 |
R4-IB3-P9A to R5-U1-P5A R4-IB3-P9B to R5-U1-P6A |
5 meters |
|
R4 IB2 within Rack 4 |
R4-IB2-P8A to R4-U1-P3B R4-IB2-P8B to R4-U1-P4B |
3 meters |
|
R4 IB2 to Rack 1 |
R4-IB2-P10A to R1-U1-P7B R4-IB2-P10B to R1-U1-P8B |
10 meters |
|
R4 IB2 to Rack 2 |
R4-IB2-P11A to R2-U1-P9B |
5 meters |
|
R4 IB2 to Rack 3 |
R4-IB2-P11B to R3-U1-P10B |
5 meters |
|
R4 IB2 to Rack 5 |
R4-IB2-P9A to R5-U1-P5B R4-IB2-P9B to R5-U1-P6B |
5 meters |
Table 6-14 shows the cable connections for the fifth spine switch (R5-U1) when cabling five full racks together:
Table 6-14 Leaf Switch Connections for the Fifth Rack in a Five-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R5 IB3 within Rack 5 |
R5-IB3-P8A to R5-U1-P3A R5-IB3-P8B to R5-U1-P4A |
3 meters |
|
R5 IB3 to Rack 1 |
R5-IB3-P9A to R1-U1-P5A R5-IB3-P9B to R1-U1-P6A |
10 meters |
|
R5 IB3 to Rack 2 |
R5-IB3-P10A to R2-U1-P7A R5-IB3-P10B to R2-U1-P8A |
10 meters |
|
R5 IB3 to Rack 3 |
R5-IB3-P11A to R3-U1-P9A |
5 meters |
|
R5 IB3 to Rack 4 |
R5-IB3-P11B to R4-U1-P10A |
5 meters |
|
R5 IB2 within Rack 5 |
R5-IB2-P8A to R5-U1-P3B R5-IB2-P8B to R5-U1-P4B |
3 meters |
|
R5 IB2 to Rack 1 |
R5-IB2-P9A to R1-U1-P5B R5-IB2-P9B to R1-U1-P6B |
10 meters |
|
R5 IB2 to Rack 2 |
R5-IB2-P10A to R2-U1-P7B R5-IB2-P10B to R2-U1-P8B |
10 meters |
|
R5 IB2 to Rack 3 |
R5-IB2-P11A to R3-U1-P9B |
5 meters |
|
R5 IB2 to Rack 4 |
R5-IB2-P11B to R4-U1-P10B |
5 meters |
6.6 Six-Rack Cabling with InfiniBand Network Fabric
Note the following when cabling six racks together with InfiniBand Network Fabric:
-
The spine switch is in U1 (also referred to as IB1) for all racks.
-
In the following tables, the leaf switches are referred to as IB2 and IB3. Use the rack unit that is appropriate for your system.
-
In Oracle Exadata Database Machine X2-2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers, the leaf switches are in U20 and U24, also referred to as IB2 and IB3, respectively.
-
In Oracle Exadata Database Machine X2-8 and later 8-socket racks, the leaf switches are in U21 and U23, also referred to as IB2 and IB3, respectively.
-
In Oracle Exadata Database Machine X4-2 and later two-socket racks, or Oracle Exadata Database Machine X3-2 Full Racks and Oracle Exadata Storage Expansion Rack X3-2 racks, the leaf switches are in U20 and U22, also referred to as IB2 and IB3, respectively.
-
The cable lengths shown in the tables assume the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cables lengths. Up to 100 meters is supported.
-
Only optical cables are supported for lengths greater than 5 meters.
Table 6-15 shows the cable connections for the first spine switch (R1-U1) when cabling six racks together:
Table 6-15 Leaf Switch Connections for the First Rack in a Six-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R1 IB3 within Rack 1 |
R1-IB3-P8A to R1-U1-P3A R1-IB3-P8B to R1-U1-P4A |
3 meters |
|
R1 IB3 to Rack 2 |
R1-IB3-P9A to R2-U1-P5A R1-IB3-P9B to R2-U1-P6A |
5 meters |
|
R1 IB3 to Rack 3 |
R1-IB3-P10A to R3-U1-P7A |
5 meters |
|
R1 IB3 to Rack 4 |
R1-IB3-P10B to R4-U1-P8A |
10 meters |
|
R1 IB3 to Rack 5 |
R1-IB3-P11A to R5-U1-P9A |
10 meters |
|
R1 IB3 to Rack 6 |
R1-IB3-P11B to R6-U1-P10A |
10 meters |
|
R1 IB2 within Rack 1 |
R1-IB2-P8A to R1-U1-P3B R1-IB2-P8B to R1-U1-P4B |
3 meters |
|
R1 IB2 to Rack 2 |
R1-IB2-P9A to R2-U1-P5B R1-IB2-P9B to R2-U1-P6B |
5 meters |
|
R1 IB2 to Rack 3 |
R1-IB2-P10A to R3-U1-P7B |
5 meters |
|
R1 IB2 to Rack 4 |
R1-IB2-P10B to R4-U1-P8B |
10 meters |
|
R1 IB2 to Rack 5 |
R1-IB2-P11A to R5-U1-P9B |
10 meters |
|
R1 IB2 to Rack 6 |
R1-IB2-P11B to R6-U1-P10B |
10 meters |
Table 6-16 shows the cable connections for the second spine switch (R2-U1) when cabling six full racks together:
Table 6-16 Leaf Switch Connections for the Second Rack in a Six-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R2 IB3 within Rack 2 |
R2-IB3-P8A to R2-U1-P3A R2-IB3-P8B to R2-U1-P4A |
3 meters |
|
R2 IB3 to Rack 1 |
R2-IB3-P11B to R1-U1-P10A |
5 meters |
|
R2 IB3 to Rack 3 |
R2-IB3-P9A to R3-U1-P5A R2-IB3-P9B to R3-U1-P6A |
5 meters |
|
R2 IB3 to Rack 4 |
R2-IB3-P10A to R4-U1-P7A |
5 meters |
|
R2 IB3 to Rack 5 |
R2-IB3-P10B to R5-U1-P8A |
10 meters |
|
R2 IB3 to Rack 6 |
R2-IB3-P11Ato R6-U1-P9A |
10 meters |
|
R2 IB2 within Rack 2 |
R2-IB2-P8A to R2-U1-P3B R2-IB2-P8B to R2-U1-P4B |
3 meters |
|
R2 IB2 to Rack 1 |
R2-IB2-P11B to R1-U1-P10B |
5 meters |
|
R2 IB2 to Rack 3 |
R2-IB2-P9A to R3-U1-P5B R2-IB2-P9B to R3-U1-P6B |
5 meters |
|
R2 IB2 to Rack 4 |
R2-IB2-P10A to R4-U1-P7B |
5 meters |
|
R2 IB2 to Rack 5 |
R2-IB2-P10B to R5-U1-P8B |
10 meters |
|
R2 IB2 to Rack 6 |
R2-IB2-P11Ato R6-U1-P9B |
10 meters |
Table 6-17 shows the cable connections for the third spine switch (R3-U1) when cabling six full racks together:
Table 6-17 Leaf Switch Connections for the Third Rack in a Six-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R3 IB3 within Rack 3 |
R3-IB3-P8A to R3-U1-P3A R3-IB3-P8B to R3-U1-P4A |
3 meters |
|
R3 IB3 to Rack 1 |
R3-IB3-P11A to R1-U1-P9A |
5 meters |
|
R3 IB3 to Rack 2 |
R3-IB3-P11B to R2-U1-P10A |
5 meters |
|
R3 IB3 to Rack 4 |
R3-IB3-P9A to R4-U1-P5A R3-IB3-P9B to R4-U1-P6A |
5 meters |
|
R3 IB3 to Rack 5 |
R3-IB3-P10A to R5-U1-P7A |
5 meters |
|
R3 IB3 to Rack 6 |
R3-IB3-P10B to R6-U1-P8A |
10 meters |
|
R3 IB2 within Rack 3 |
R3-IB2-P8A to R3-U1-P3B R3-IB2-P8B to R3-U1-P4B |
3 meters |
|
R3 IB2 to Rack 1 |
R3-IB2-P11A to R1-U1-P9B |
5 meters |
|
R3 IB2 to Rack 2 |
R3-IB2-P11B to R2-U1-P10B |
5 meters |
|
R3 IB2 to Rack 4 |
R3-IB2-P9A to R4-U1-P5B R3-IB2-P9B to R4-U1-P6B |
5 meters |
|
R3 IB2 to Rack 5 |
R3-IB2-P10A to R5-U1-P7B |
5 meters |
|
R3 IB2 to Rack 6 |
R3-IB2-P10B to R6-U1-P8B |
10 meters |
Table 6-18 shows the cable connections for the fourth spine switch (R4-U1) when cabling six full racks together:
Table 6-18 Leaf Switch Connections for the Fourth Rack in a Six-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R4 IB3 within Rack 4 |
R4-IB3-P8A to R4-U1-P3A R4-IB3-P8B to R4-U1-P4A |
3 meters |
|
R4 IB3 to Rack 1 |
R4-IB3-P10B to R1-U1-P8A |
10 meters |
|
R4 IB3 to Rack 2 |
R4-IB3-P11A to R2-U1-P9A |
5 meters |
|
R4 IB3 to Rack 3 |
R4-IB3-P11B to R3-U1-P10A |
5 meters |
|
R4 IB3 to Rack 5 |
R4-IB3-P9A to R5-U1-P5A R4-IB3-P9B to R5-U1-P6A |
5 meters |
|
R4 IB3 to Rack 6 |
R4-IB3-P10A to R6-U1-P7A |
5 meters |
|
R4 IB2 within Rack 4 |
R4-IB2-P8A to R4-U1-P3B R4-IB2-P8B to R4-U1-P4B |
3 meters |
|
R4 IB2 to Rack 1 |
R4-IB2-P10B to R1-U1-P8B |
10 meters |
|
R4 IB2 to Rack 2 |
R4-IB2-P11A to R2-U1-P9B |
5 meters |
|
R4 IB2 to Rack 3 |
R4-IB2-P11B to R3-U1-P10B |
5 meters |
|
R4 IB2 to Rack 5 |
R4-IB2-P9A to R5-U1-P5B R4-IB2-P9B to R5-U1-P6B |
5 meters |
|
R4 IB2 to Rack 6 |
R4-IB2-P10A to R6-U1-P7B |
5 meters |
Table 6-19 shows the cable connections for the fifth spine switch (R5-U1) when cabling six full racks together:
Table 6-19 Leaf Switch Connections for the Fifth Rack in a Six-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R5 IB3 within Rack 5 |
R5-IB3-P8A to R5-U1-P3A R5-IB3-P8B to R5-U1-P4A |
3 meters |
|
R5 IB3 to Rack 1 |
R5-IB3-P10A to R1-U1-P7A |
10 meters |
|
R5 IB3 to Rack 2 |
R5-IB3-P10B to R2-U1-P8A |
10 meters |
|
R5 IB3 to Rack 3 |
R5-IB3-P11A to R3-U1-P9A |
5 meters |
|
R5 IB3 to Rack 4 |
R5-IB3-P11B to R4-U1-P10A |
5 meters |
|
R5 IB3 to Rack 6 |
R5-IB3-P9A to R6-U1-P5A R5-IB3-P9B to R6-U1-P6A |
5 meters |
|
R5 IB2 within Rack 5 |
R5-IB2-P8A to R5-U1-P3B R5-IB2-P8B to R5-U1-P4B |
3 meters |
|
R5 IB2 to Rack 1 |
R5-IB2-P10A to R1-U1-P7B |
10 meters |
|
R5 IB2 to Rack 2 |
R5-IB2-P10B to R2-U1-P8B |
10 meters |
|
R5 IB2 to Rack 3 |
R5-IB2-P11A to R3-U1-P9B |
5 meters |
|
R5 IB2 to Rack 4 |
R5-IB2-P11B to R4-U1-P10B |
5 meters |
|
R5 IB2 to Rack 6 |
R5-IB2-P9A to R6-U1-P5B R5-IB2-P9B to R6-U1-P6B |
5 meters |
Table 6-20 shows the cable connections for the sixth spine switch (R6-U1) when cabling six full racks together:
Table 6-20 Leaf Switch Connections for the Sixth Rack in a Six-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R6 IB3 within Rack 6 |
R6-IB3-P8A to R6-U1-P3A R6-IB3-P8B to R6-U1-P4A |
3 meters |
|
R6 IB3 to Rack 1 |
R6-IB3-P9A to R1-U1-P5A R6-IB3-P9B to R1-U1-P6A |
10 meters |
|
R6 IB3 to Rack 2 |
R6-IB3-P10A to R2-U1-P7A |
10 meters |
|
R6 IB3 to Rack 3 |
R6-IB3-P10B to R3-U1-P8A |
10 meters |
|
R6 IB3 to Rack 4 |
R6-IB3-P11A to R4-U1-P9A |
5 meters |
|
R6 IB3 to Rack 5 |
R6-IB3-P11B to R5-U1-P10A |
5 meters |
|
R6 IB2 within Rack 6 |
R6-IB2-P8A to R6-U1-P3B R6-IB2-P8B to R6-U1-P4B |
3 meters |
|
R6 IB2 to Rack 1 |
R6-IB2-P9A to R1-U1-P5B R6-IB2-P9B to R1-U1-P6B |
10 meters |
|
R6 IB2 to Rack 2 |
R6-IB2-P10A to R2-U1-P7B |
10 meters |
|
R6 IB2 to Rack 3 |
R6-IB2-P10B to R3-U1-P8B |
10 meters |
|
R6 IB2 to Rack 4 |
R6-IB2-P11A to R4-U1-P9B |
5 meters |
|
R6 IB2 to Rack 5 |
R6-IB2-P11B to R5-U1-P10B |
5 meters |
6.7 Seven-Rack Cabling with InfiniBand Network Fabric
Note the following when cabling seven racks together with InfiniBand Network Fabric:
-
The spine switch is in U1 (also referred to as IB1) for all racks.
-
In the following tables, the leaf switches are referred to as IB2 and IB3. Use the rack unit that is appropriate for your system.
-
In Oracle Exadata Database Machine X2-2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers, the leaf switches are in U20 and U24, also referred to as IB2 and IB3, respectively.
-
In Oracle Exadata Database Machine X2-8 and later racks, the leaf switches are in U21 and U23, also referred to as IB2 and IB3, respectively.
-
In Oracle Exadata Database Machine X4-2 and later racks, or Oracle Exadata Database Machine X3-2 Full Racks and Oracle Exadata Storage Expansion Rack X3-2 racks, the leaf switches are in U20 and U22, also referred to as IB2 and IB3, respectively.
-
The cable lengths shown in the tables assume the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cables lengths. Up to 100 meters is supported.
-
Only optical cables are supported for lengths greater than 5 meters.
Table 6-21 shows the cable connections for the first spine switch (R1-U1) when cabling seven racks together:
Table 6-21 Leaf Switch Connections for the First Rack in a Seven-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R1 IB3 within Rack 1 |
R1-IB3-P8A to R1-U1-P3A R1-IB3-P8B to R1-U1-P4A |
3 meters |
|
R1 IB3 to Rack 2 |
R1-IB3-P9A to R2-U1-P5A |
5 meters |
|
R1 IB3 to Rack 3 |
R1-IB3-P9B to R3-U1-P6A |
5 meters |
|
R1 IB3 to Rack 4 |
R1-IB3-P10A to R4-U1-P7A |
10 meters |
|
R1 IB3 to Rack 5 |
R1-IB3-P10B to R5-U1-P8A |
10 meters |
|
R1 IB3 to Rack 6 |
R1-IB3-P11A to R6-U1-P9A |
10 meters |
|
R1 IB3 to Rack 7 |
R1-IB3-P11B to R7-U1-P10A |
10 meters |
|
R1 IB2 within Rack 1 |
R1-IB2-P8A to R1-U1-P3B R1-IB2-P8B to R1-U1-P4B |
3 meters |
|
R1 IB2 to Rack 2 |
R1-IB2-P9A to R2-U1-P5B |
5 meters |
|
R1 IB2 to Rack 3 |
R1-IB2-P9B to R3-U1-P6B |
5 meters |
|
R1 IB2 to Rack 4 |
R1-IB2-P10A to R4-U1-P7B |
10 meters |
|
R1 IB2 to Rack 5 |
R1-IB2-P10B to R5-U1-P8B |
10 meters |
|
R1 IB2 to Rack 6 |
R1-IB2-P11A to R6-U1-P9B |
10 meters |
|
R1 IB2 to Rack 7 |
R1-IB2-P11B to R7-U1-P10B |
10 meters |
Table 6-22 shows the cable connections for the second spine switch (R2-U1) when cabling seven full racks together:
Table 6-22 Leaf Switch Connections for the Second Rack in a Seven-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R2 IB3 within Rack 2 |
R2-IB3-P8A to R2-U1-P3A R2-IB3-P8B to R2-U1-P4A |
3 meters |
|
R2 IB3 to Rack 1 |
R2-IB3-P11B to R1-U1-P10A |
5 meters |
|
R2 IB3 to Rack 3 |
R2-IB3-P9A to R3-U1-P5A |
5 meters |
|
R2 IB3 to Rack 4 |
R2-IB3-P9B to R4-U1-P6A |
5 meters |
|
R2 IB3 to Rack 5 |
R2-IB3-P10A to R5-U1-P7A |
10 meters |
|
R2 IB3 to Rack 6 |
R2-IB3-P10B to R6-U1-P8A |
10 meters |
|
R2 IB3 to Rack 7 |
R2-IB3-P11A to R7-U1-P9A |
10 meters |
|
R2 IB2 within Rack 2 |
R2-IB2-P8A to R2-U1-P3B R2-IB2-P8B to R2-U1-P4B |
3 meters |
|
R2 IB2 to Rack 1 |
R2-IB2-P11B to R1-U1-P10B |
5 meters |
|
R2 IB2 to Rack 3 |
R2-IB2-P9A to R3-U1-P5B |
5 meters |
|
R2 IB2 to Rack 4 |
R2-IB2-P9B to R4-U1-P6B |
5 meters |
|
R2 IB2 to Rack 5 |
R2-IB2-P10A to R5-U1-P7B |
10 meters |
|
R2 IB2 to Rack 6 |
R2-IB2-P10Bto R6-U1-P8B |
10 meters |
|
R2 IB2 to Rack 7 |
R2-IB2-P11A to R7-U1-P9B |
10 meters |
Table 6-23 shows the cable connections for the third spine switch (R3-U1) when cabling seven full racks together:
Table 6-23 Leaf Switch Connections for the Third Rack in a Seven-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R3 IB3 within Rack 3 |
R3-IB3-P8A to R3-U1-P3A R3-IB3-P8B to R3-U1-P4A |
3 meters |
|
R3 IB3 to Rack 1 |
R3-IB3-P11A to R1-U1-P9A |
5 meters |
|
R3 IB3 to Rack 2 |
R3-IB3-P11B to R2-U1-P10A |
5 meters |
|
R3 IB3 to Rack 4 |
R3-IB3-P9A to R4-U1-P5A |
5 meters |
|
R3 IB3 to Rack 5 |
R3-IB3-P9B to R5-U1-P6A |
5 meters |
|
R3 IB3 to Rack 6 |
R3-IB3-P10A to R6-U1-P7A |
10 meters |
|
R3 IB3 to Rack 7 |
R3-IB3-P10B to R7-U1-P8A |
10 meters |
|
R3 IB2 within Rack 3 |
R3-IB2-P8A to R3-U1-P3B R3-IB2-P8B to R3-U1-P4B |
3 meters |
|
R3 IB2 to Rack 1 |
R3-IB2-P11A to R1-U1-P9B |
5 meters |
|
R3 IB2 to Rack 2 |
R3-IB2-P11B to R2-U1-P10B |
5 meters |
|
R3 IB2 to Rack 4 |
R3-IB2-P9A to R4-U1-P5B |
5 meters |
|
R3 IB2 to Rack 5 |
R3-IB2-P9B to R5-U1-P6B |
5 meters |
|
R3 IB2 to Rack 6 |
R3-IB2-P10A to R6-U1-P7B |
10 meters |
|
R3 IB2 to Rack 7 |
R3-IB2-P10B to R7-U1-P8B |
10 meters |
Table 6-24 shows the cable connections for the fourth spine switch (R4-U1) when cabling seven full racks together:
Table 6-24 Leaf Switch Connections for the Fourth Rack in a Seven-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R4 IB3 within Rack 4 |
R4-IB3-P8A to R4-U1-P3A R4-IB3-P8B to R4-U1-P4A |
3 meters |
|
R4 IB3 to Rack 1 |
R4-IB3-P10B to R1-U1-P8A |
10 meters |
|
R4 IB3 to Rack 2 |
R4-IB3-P11A to R2-U1-P9A |
5 meters |
|
R4 IB3 to Rack 3 |
R4-IB3-P11B to R3-U1-P10A |
5 meters |
|
R4 IB3 to Rack 5 |
R4-IB3-P9A to R5-U1-P5A |
5 meters |
|
R4 IB3 to Rack 6 |
R4-IB3-P9B to R6-U1-P6A |
5 meters |
|
R4 IB3 to Rack 7 |
R4-IB3-P10A to R7-U1-P7A |
10 meters |
|
R4 IB2 within Rack 4 |
R4-IB2-P8A to R4-U1-P3B R4-IB2-P8B to R4-U1-P4B |
3 meters |
|
R4 IB2 to Rack 1 |
R4-IB2-P10B to R1-U1-P8B |
10 meters |
|
R4 IB2 to Rack 2 |
R4-IB2-P11A to R2-U1-P9B |
5 meters |
|
R4 IB2 to Rack 3 |
R4-IB2-P11B to R3-U1-P10B |
5 meters |
|
R4 IB2 to Rack 5 |
R4-IB2-P9A to R5-U1-P5B |
5 meters |
|
R4 IB2 to Rack 6 |
R4-IB2-P9B to R6-U1-P6B |
5 meters |
|
R4 IB2 to Rack 7 |
R4-IB2-P10A to R7-U1-P7B |
10 meters |
Table 6-25 shows the cable connections for the fifth spine switch (R5-U1) when cabling seven full racks together:
Table 6-25 Leaf Switch Connections for the Fifth Rack in a Seven-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R5 IB3 within Rack 5 |
R5-IB3-P8A to R5-U1-P3A R5-IB3-P8B to R5-U1-P4A |
3 meters |
|
R5 IB3 to Rack 1 |
R5-IB3-P10A to R1-U1-P7A |
10 meters |
|
R5 IB3 to Rack 2 |
R5-IB3-P10B to R2-U1-P8A |
10 meters |
|
R5 IB3 to Rack 3 |
R5-IB3-P11A to R3-U1-P9A |
5 meters |
|
R5 IB3 to Rack 4 |
R5-IB3-P11B to R4-U1-P10A |
5 meters |
|
R5 IB3 to Rack 6 |
R5-IB3-P9A to R6-U1-P5A |
5 meters |
|
R5 IB3 to Rack 7 |
R5-IB3-P9B to R7-U1-P6A |
5 meters |
|
R5 IB2 within Rack 5 |
R5-IB2-P8A to R5-U1-P3B R5-IB2-P8B to R5-U1-P4B |
3 meters |
|
R5 IB2 to Rack 1 |
R5-IB2-P10A to R1-U1-P7B |
10 meters |
|
R5 IB2 to Rack 2 |
R5-IB2-P10B to R2-U1-P8B |
10 meters |
|
R5 IB2 to Rack 3 |
R5-IB2-P11A to R3-U1-P9B |
5 meters |
|
R5 IB2 to Rack 4 |
R5-IB2-P11B to R4-U1-P10B |
5 meters |
|
R5 IB2 to Rack 6 |
R5-IB2-P9A to R6-U1-P5B |
5 meters |
|
R5 IB2 to Rack 7 |
R5-IB2-P9B to R7-U1-P6B |
5 meters |
Table 6-26 shows the cable connections for the sixth spine switch (R6-U1) when cabling seven full racks together:
Table 6-26 Leaf Switch Connections for the Sixth Rack in a Seven-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R6 IB3 within Rack 6 |
R6-IB3-P8A to R6-U1-P3A R6-IB3-P8B to R6-U1-P4A |
3 meters |
|
R6 IB3 to Rack 1 |
R6-IB3-P9B to R1-U1-P6A |
10 meters |
|
R6 IB3 to Rack 2 |
R6-IB3-P10A to R2-U1-P7A |
10 meters |
|
R6 IB3 to Rack 3 |
R6-IB3-P10B to R3-U1-P8A |
10 meters |
|
R6 IB3 to Rack 4 |
R6-IB3-P11A to R4-U1-P9A |
5 meters |
|
R6 IB3 to Rack 5 |
R6-IB3-P11B to R5-U1-P10A |
5 meters |
|
R6 IB3 to Rack 7 |
R6-IB3-P9A to R7-U1-P5A |
5 meters |
|
R6 IB2 within Rack 6 |
R6-IB2-P8A to R6-U1-P3B R6-IB2-P8B to R6-U1-P4B |
3 meters |
|
R6 IB2 to Rack 1 |
R6-IB2-P9B to R1-U1-P6B |
10 meters |
|
R6 IB2 to Rack 2 |
R6-IB2-P10A to R2-U1-P7B |
10 meters |
|
R6 IB2 to Rack 3 |
R6-IB2-P10B to R3-U1-P8B |
10 meters |
|
R6 IB2 to Rack 4 |
R6-IB2-P11A to R4-U1-P9B |
5 meters |
|
R6 IB2 to Rack 5 |
R6-IB2-P11B to R5-U1-P10B |
5 meters |
|
R6 IB2 to Rack 7 |
R6-IB2-P9A to R7-U1-P5B |
5 meters |
Table 6-27 shows the cable connections for the seventh spine switch (R7-U1) when cabling seven full racks together:
Table 6-27 Leaf Switch Connections for the Seventh Rack in a Seven-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R7 IB3 within Rack 7 |
R7-IB3-P8A to R7-U1-P3A R7-IB3-P8B to R7-U1-P4A |
3 meters |
|
R7 IB3 to Rack 1 |
R7-IB3-P9A to R1-U1-P5A |
10 meters |
|
R7 IB3 to Rack 2 |
R7-IB3-P9B to R2-U1-P6A |
10 meters |
|
R7 IB3 to Rack 3 |
R7-IB3-P10A to R3-U1-P7A |
10 meters |
|
R7 IB3 to Rack 4 |
R7-IB3-P10B to R4-U1-P8A |
10 meters |
|
R7 IB3 to Rack 5 |
R7-IB3-P11A to R5-U1-P9A |
5 meters |
|
R7 IB3 to Rack 6 |
R7-IB3-P11B to R6-U1-P10A |
5 meters |
|
R7 IB2 within Rack 7 |
R7-IB2-P8A to R7-U1-P3B R7-IB2-P8B to R7-U1-P4B |
3 meters |
|
R7 IB2 to Rack 1 |
R7-IB2-P9A to R1-U1-P5B |
10 meters |
|
R7 IB2 to Rack 2 |
R7-IB2-P9B to R2-U1-P6B |
10 meters |
|
R7 IB2 to Rack 3 |
R7-IB2-P10A to R3-U1-P7B |
10 meters |
|
R7 IB2 to Rack 4 |
R7-IB2-P10B to R4-U1-P8B |
10 meters |
|
R7 IB2 to Rack 5 |
R7-IB2-P11A to R5-U1-P9B |
5 meters |
|
R7 IB2 to Rack 6 |
R7-IB2-P11B to R6-U1-P10B |
5 meters |
6.8 Eight-Rack Cabling with InfiniBand Network Fabric
Note the following when cabling eight racks together with InfiniBand Network Fabric:
-
The spine switch is in U1 (also referred to as IB1) for all racks.
-
In the following tables, the leaf switches are referred to as IB2 and IB3. Use the rack unit that is appropriate for your system.
-
In Oracle Exadata Database Machine X2-2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers, the leaf switches are in U20 and U24, also referred to as IB2 and IB3, respectively.
-
In Oracle Exadata Database Machine X2-8 and later racks, the leaf switches are in U21 and U23, also referred to as IB2 and IB3, respectively.
-
In Oracle Exadata Database Machine X4-2 and later racks, or Oracle Exadata Database Machine X3-2 Full Racks and Oracle Exadata Storage Expansion Rack X3-2 racks, the leaf switches are in U20 and U22, also referred to as IB2 and IB3, respectively.
-
The cable lengths shown in the tables assume the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cables lengths. Up to 100 meters is supported.
-
Only optical cables are supported for lengths greater than 5 meters.
Table 6-28 shows the cable connections for the first spine switch (R1-U1) when cabling eight racks together:
Table 6-28 Leaf Switch Connections for the First Rack in a Eight-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R1 IB3 within Rack 1 |
R1-IB3-P8A to R1-U1-P3A |
3 meters |
|
R1 IB3 to Rack 2 |
R1-IB3-P8B to R2-U1-P4A |
5 meters |
|
R1 IB3 to Rack 3 |
R1-IB3-P9A to R3-U1-P5A |
5 meters |
|
R1 IB3 to Rack 4 |
R1-IB3-P9B to R4-U1-P6A |
10 meters |
|
R1 IB3 to Rack 5 |
R1-IB3-P10A to R5-U1-P7A |
10 meters |
|
R1 IB3 to Rack 6 |
R1-IB3-P10B to R6-U1-P8A |
10 meters |
|
R1 IB3 to Rack 7 |
R1-IB3-P11A to R7-U1-P9A |
10 meters |
|
R1 IB3 to Rack 8 |
R1-IB3-P11B to R8-U1-P10A |
10 meters |
|
R1 IB2 within Rack 1 |
R1-IB2-P8A to R1-U1-P3B |
3 meters |
|
R1 IB2 to Rack 2 |
R1-IB2-P8B to R2-U1-P4B |
5 meters |
|
R1 IB2 to Rack 3 |
R1-IB2-P9A to R3-U1-P5B |
5 meters |
|
R1 IB2 to Rack 4 |
R1-IB2-P9B to R4-U1-P6B |
10 meters |
|
R1 IB2 to Rack 5 |
R1-IB2-P10A to R5-U1-P7B |
10 meters |
|
R1 IB2 to Rack 6 |
R1-IB2-P10B to R6-U1-P8B |
10 meters |
|
R1 IB2 to Rack 7 |
R1-IB2-P11A to R7-U1-P9B |
10 meters |
|
R1 IB2 to Rack 8 |
R1-IB2-P11B to R8-U1-P10B |
10 meters |
Table 6-29 shows the cable connections for the second spine switch (R2-U1) when cabling eight full racks together:
Table 6-29 Leaf Switch Connections for the Second Rack in a Eight-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R2 IB3 within Rack 2 |
R2-IB3-P8A to R2-U1-P3A |
3 meters |
|
R2 IB3 to Rack 1 |
R2-IB3-P11B to R1-U1-P10A |
5 meters |
|
R2 IB3 to Rack 3 |
R2-IB3-P8B to R3-U1-P4A |
5 meters |
|
R2 IB3 to Rack 4 |
R2-IB3-P9A to R4-U1-P5A |
5 meters |
|
R2 IB3 to Rack 5 |
R2-IB3-P9B to R5-U1-P6A |
10 meters |
|
R2 IB3 to Rack 6 |
R2-IB3-P10A to R6-U1-P7A |
10 meters |
|
R2 IB3 to Rack 7 |
R2-IB3-P10B to R7-U1-P8A |
10 meters |
|
R2 IB3 to Rack 8 |
R2-IB3-P11A to R8-U1-P9A |
10 meters |
|
R2 IB2 within Rack 2 |
R2-IB2-P8A to R2-U1-P3B |
3 meters |
|
R2 IB2 to Rack 1 |
R2-IB2-P11B to R1-U1-P10B |
5 meters |
|
R2 IB2 to Rack 3 |
R2-IB2-P8B to R3-U1-P4B |
5 meters |
|
R2 IB2 to Rack 4 |
R2-IB2-P9A to R4-U1-P5B |
5 meters |
|
R2 IB2 to Rack 5 |
R2-IB2-P9B to R5-U1-P6B |
10 meters |
|
R2 IB2 to Rack 6 |
R2-IB2-P10A to R6-U1-P7B |
10 meters |
|
R2 IB2 to Rack 7 |
R2-IB2-P10B to R7-U1-P8B |
10 meters |
|
R2 IB2 to Rack 8 |
R2-IB2-P11A to R8-U1-P9B |
10 meters |
Table 6-30 shows the cable connections for the third spine switch (R3-U1) when cabling eight full racks together:
Table 6-30 Leaf Switch Connections for the Third Rack in a Eight-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R3 IB3 within Rack 3 |
R3-IB3-P8A to R3-U1-P3A |
3 meters |
|
R3 IB3 to Rack 1 |
R3-IB3-P11A to R1-U1-P9A |
5 meters |
|
R3 IB3 to Rack 2 |
R3-IB3-P11B to R2-U1-P10A |
5 meters |
|
R3 IB3 to Rack 4 |
R3-IB3-P8B to R4-U1-P4A |
5 meters |
|
R3 IB3 to Rack 5 |
R3-IB3-P9A to R5-U1-P5A |
5 meters |
|
R3 IB3 to Rack 6 |
R3-IB3-P9B to R6-U1-P6A |
10 meters |
|
R3 IB3 to Rack 7 |
R3-IB3-P10A to R7-U1-P7A |
10 meters |
|
R3 IB3 to Rack 8 |
R3-IB3-P10B to R8-U1-P8A |
10 meters |
|
R3 IB2 within Rack 3 |
R3-IB2-P8A to R3-U1-P3B |
3 meters |
|
R3 IB2 to Rack 1 |
R3-IB2-P11A to R1-U1-P9B |
5 meters |
|
R3 IB2 to Rack 2 |
R3-IB2-P11B to R2-U1-P10B |
5 meters |
|
R3 IB2 to Rack 4 |
R3-IB2-P8B to R4-U1-P4B |
5 meters |
|
R3 IB2 to Rack 5 |
R3-IB2-P9A to R5-U1-P5B |
5 meters |
|
R3 IB2 to Rack 6 |
R3-IB2-P9B to R6-U1-P6B |
10 meters |
|
R3 IB2 to Rack 7 |
R3-IB2-P10A to R7-U1-P7B |
10 meters |
|
R3 IB2 to Rack 8 |
R3-IB2-P10B to R8-U1-P8B |
10 meters |
Table 6-31 shows the cable connections for the fourth spine switch (R4-U1) when cabling eight full racks together:
Table 6-31 Leaf Switch Connections for the Fourth Rack in a Eight-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R4 IB3 within Rack 4 |
R4-IB3-P8A to R4-U1-P3A |
3 meters |
|
R4 IB3 to Rack 1 |
R4-IB3-P10B to R1-U1-P8A |
10 meters |
|
R4 IB3 to Rack 2 |
R4-IB3-P11A to R2-U1-P9A |
5 meters |
|
R4 IB3 to Rack 3 |
R4-IB3-P11B to R3-U1-P10A |
5 meters |
|
R4 IB3 to Rack 5 |
R4-IB3-P8B to R5-U1-P4A |
5 meters |
|
R4 IB3 to Rack 6 |
R4-IB3-P9A to R6-U1-P5A |
5 meters |
|
R4 IB3 to Rack 7 |
R4-IB3-P9B to R7-U1-P6A |
10 meters |
|
R4 IB3 to Rack 8 |
R4-IB3-P10A to R8-U1-P7A |
10 meters |
|
R4 IB2 within Rack 4 |
R4-IB2-P8A to R4-U1-P3B |
3 meters |
|
R4 IB2 to Rack 1 |
R4-IB2-P10B to R1-U1-P8B |
10 meters |
|
R4 IB2 to Rack 2 |
R4-IB2-P11A to R2-U1-P9B |
5 meters |
|
R4 IB2 to Rack 3 |
R4-IB2-P11B to R3-U1-P10B |
5 meters |
|
R4 IB2 to Rack 5 |
R4-IB2-P8B to R5-U1-P4B |
5 meters |
|
R4 IB2 to Rack 6 |
R4-IB2-P9A to R6-U1-P5B |
5 meters |
|
R4 IB2 to Rack 7 |
R4-IB2-P9B to R7-U1-P6B |
10 meters |
|
R4 IB2 to Rack 8 |
R4-IB2-P10A to R8-U1-P7B |
10 meters |
Table 6-32 shows the cable connections for the fifth spine switch (R5-U1) when cabling eight full racks together:
Table 6-32 Leaf Switch Connections for the Fifth Rack in a Eight-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R5 IB3 within Rack 5 |
R5-IB3-P8A to R5-U1-P3A |
3 meters |
|
R5 IB3 to Rack 1 |
R5-IB3-P10A to R1-U1-P7A |
10 meters |
|
R5 IB3 to Rack 2 |
R5-IB3-P10B to R2-U1-P8A |
10 meters |
|
R5 IB3 to Rack 3 |
R5-IB3-P11A to R3-U1-P9A |
5 meters |
|
R5 IB3 to Rack 4 |
R5-IB3-P11B to R4-U1-P10A |
5 meters |
|
R5 IB3 to Rack 6 |
R5-IB3-P8B to R6-U1-P4A |
5 meters |
|
R5 IB3 to Rack 7 |
R5-IB3-P9A to R7-U1-P5A |
5 meters |
|
R5 IB3 to Rack 8 |
R5-IB3-P9B to R8-U1-P6A |
10 meters |
|
R5 IB2 within Rack 5 |
R5-IB2-P8A to R5-U1-P3B |
3 meters |
|
R5 IB2 to Rack 1 |
R5-IB2-P10A to R1-U1-P7B |
10 meters |
|
R5 IB2 to Rack 2 |
R5-IB2-P10B to R2-U1-P8B |
10 meters |
|
R5 IB2 to Rack 3 |
R5-IB2-P11A to R3-U1-P9B |
5 meters |
|
R5 IB2 to Rack 4 |
R5-IB2-P11B to R4-U1-P10B |
5 meters |
|
R5 IB2 to Rack 6 |
R5-IB2-P8B to R6-U1-P4B |
5 meters |
|
R5 IB2 to Rack 7 |
R5-IB2-P9A to R7-U1-P5B |
5 meters |
|
R5 IB2 to Rack 8 |
R5-IB2-P9B to R8-U1-P6B |
10 meters |
Table 6-33 shows the cable connections for the sixth spine switch (R6-U1) when cabling eight full racks together:
Table 6-33 Leaf Switch Connections for the Sixth Rack in a Eight-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R6 IB3 within Rack 6 |
R6-IB3-P8A to R6-U1-P3A |
3 meters |
|
R6 IB3 to Rack 1 |
R6-IB3-P9B to R1-U1-P6A |
10 meters |
|
R6 IB3 to Rack 2 |
R6-IB3-P10A to R2-U1-P7A |
10 meters |
|
R6 IB3 to Rack 3 |
R6-IB3-P10B to R3-U1-P8A |
10 meters |
|
R6 IB3 to Rack 4 |
R6-IB3-P11A to R4-U1-P9A |
5 meters |
|
R6 IB3 to Rack 5 |
R6-IB3-P11B to R5-U1-P10A |
5 meters |
|
R6 IB3 to Rack 7 |
R6-IB3-P8B to R7-U1-P4A |
5 meters |
|
R6 IB3 to Rack 8 |
R6-IB3-P9A to R8-U1-P5A |
5 meters |
|
R6 IB2 within Rack 6 |
R6-IB2-P8A to R6-U1-P3B |
3 meters |
|
R6 IB2 to Rack 1 |
R6-IB2-P9B to R1-U1-P6B |
10 meters |
|
R6 IB2 to Rack 2 |
R6-IB3-P10A to R2-U1-P7B |
10 meters |
|
R6 IB2 to Rack 3 |
R6-IB2-P10B to R3-U1-P8B |
10 meters |
|
R6 IB2 to Rack 4 |
R6-IB2-P11A to R4-U1-P9B |
5 meters |
|
R6 IB2 to Rack 5 |
R6-IB2-P11B to R5-U1-P10B |
5 meters |
|
R6 IB2 to Rack 7 |
R6-IB2-P8B to R7-U1-P4B |
5 meters |
|
R6 IB2 to Rack 8 |
R6-IB2-P9A to R8-U1-P5B |
5 meters |
Table 6-34 shows the cable connections for the seventh spine switch (R7-U1) when cabling eight full racks together:
Table 6-34 Leaf Switch Connections for the Seventh Rack in a Eight-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R7 IB3 within Rack 7 |
R7-IB3-P8A to R7-U1-P3A |
3 meters |
|
R7 IB3 to Rack 1 |
R7-IB3-P9A to R1-U1-P5A |
10 meters |
|
R7 IB3 to Rack 2 |
R7-IB3-P9B to R2-U1-P6A |
10 meters |
|
R7 IB3 to Rack 3 |
R7-IB3-P10A to R3-U1-P7A |
10 meters |
|
R7 IB3 to Rack 4 |
R7-IB3-P10B to R4-U1-P8A |
10 meters |
|
R7 IB3 to Rack 5 |
R7-IB3-P11A to R5-U1-P9A |
5 meters |
|
R7 IB3 to Rack 6 |
R7-IB3-P11B to R6-U1-P10A |
5 meters |
|
R7 IB3 to Rack 8 |
R7-IB3-P8B to R8-U1-P4A |
5 meters |
|
R7 IB2 within Rack 7 |
R7-IB2-P8A to R7-U1-P3B |
3 meters |
|
R7 IB2 to Rack 1 |
R7-IB2-P9A to R1-U1-P5B |
10 meters |
|
R7 IB2 to Rack 2 |
R7-IB2-P9B to R2-U1-P6B |
10 meters |
|
R7 IB2 to Rack 3 |
R7-IB2-P10A to R3-U1-P7B |
10 meters |
|
R7 IB2 to Rack 4 |
R7-IB2-P10B to R4-U1-P8B |
10 meters |
|
R7 IB2 to Rack 5 |
R7-IB2-P11A to R5-U1-P9B |
5 meters |
|
R7 IB2 to Rack 6 |
R7-IB2-P11B to R6-U1-P10B |
5 meters |
|
R7 IB2 to Rack 8 |
R7-IB2-P8B to R8-U1-P4B |
5 meters |
Table 6-35 shows the cable connections for the eighth spine switch (R8-U1) when cabling eight full racks together:
Table 6-35 Leaf Switch Connections for the Eighth Rack in a Eight-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R8 IB3 within Rack 8 |
R8-IB3-P8A to R8-U1-P3A |
3 meters |
|
R8 IB3 to Rack 1 |
R8-IB3-P8B to R1-U1-P4A |
10 meters |
|
R8 IB3 to Rack 2 |
R8-IB3-P9A to R2-U1-P5A |
10 meters |
|
R8 IB3 to Rack 3 |
R8-IB3-P9B to R3-U1-P6A |
10 meters |
|
R8 IB3 to Rack 4 |
R8-IB3-P10A to R4-U1-P7A |
10 meters |
|
R8 IB3 to Rack 5 |
R8-IB3-P10B to R5-U1-P8A |
10 meters |
|
R8 IB3 to Rack 6 |
R8-IB3-P11A to R6-U1-P9A |
5 meters |
|
R8 IB3 to Rack 7 |
R8-IB3-P11B to R7-U1-P10A |
5 meters |
|
R8 IB2 within Rack 8 |
R8-IB2-P8A to R8-U1-P3B |
3 meters |
|
R8 IB2 to Rack 1 |
R8-IB2-P8B to R1-U1-P4B |
10 meters |
|
R8 IB2 to Rack 2 |
R8-IB2-P9A to R2-U1-P5B |
10 meters |
|
R8 IB2 to Rack 3 |
R8-IB2-P9B to R3-U1-P6B |
10 meters |
|
R8 IB2 to Rack 4 |
R8-IB2-P10A to R4-U1-P7B |
10 meters |
|
R8 IB2 to Rack 5 |
R8-IB2-P10B to R5-U1-P8B |
10 meters |
|
R8 IB2 to Rack 6 |
R8-IB2-P11A to R6-U1-P9B |
5 meters |
|
R8 IB2 to Rack 7 |
R8-IB2-P11B to R7-U1-P10B |
5 meters |