5 MultiRack Cabling Tables for Oracle Engineered System Rack X2 to X8
This appendix contains the multirack cabling tables for Oracle Engineered System Racks X2 to X8, which use a InfiniBand Transport Layer systems based on an InfiniBand Network Layer network.
5.1 Understanding MultiRack Cabling
Up to eight racks can be cabled together without external RDMA Network Fabric switches.
Three RDMA Network Fabric switches are needed when cabling racks together. If any rack does not include a third switch, then order the expansion switch kit. These switches attach to standard RDMA Network Fabric connectors at the end of the cables. The procedures in this section assume the racks are adjacent to each other. If they are not, then longer cables may be required for the connections.
Note:
Only for InfiniBand Transport Layer systems based on an InfiniBand Network Layer (X8 and earlier).
For Eighth or Quarter Racks, which are the smallest Elastic Configurations, follow the instructions in "Cabling Oracle Exadata Quarter Racks and Oracle Exadata Eighth Racks" for direct connection without spine switch.

For other racks (Half Rack, Full Rack, Elastic Configurations larger than Eighth or Quarter Rack) install a spine switch and follow the standard Nrack cabling using fat tree topology for two or more interconnected racks, for example "TwoRack Cabling."
The switch at rack unit 1 (U1) is referred to as the spine switch.
The following switches are referred to as leaf switches:

The switches at rack unit 20 (U20) and rack unit 24 (U24) in Oracle Engineered System X22 racks and Oracle Exadata Storage Expansion Racks

The switches at rack unit 21 (U21) and rack unit 23 (U23) in Oracle Engineered System X28 and later.

The switches at rack unit 20 (U20) and 22 (U22) in Oracle Engineered System X32 and later racks or Storage Expansion Rack X32 and later.
Note:
Oracle Exadata Database Machine X42 and later racks or Oracle Exadata Database Machine X38 Full Racks with Exadata Storage Server X42L Servers do not include spare cables or a third Sun Datacenter InfiniBand Switch 36 switch. To extend Oracle Exadata Database Machine X42 and later racks or Oracle Exadata Database Machine X38 Full Racks with Exadata Storage Server X42L Servers, you must order cables and a Sun Datacenter InfiniBand Switch 36 switch.
In a single rack, the two leaf switches are interconnected using seven connections. In addition, each leaf switch has one connection to the spine switch. The leaf switches connect to the spine switch as shown in the following graphic:
Figure 51 Connections Between Spine Switch and Leaf Switches in a Single Rack
Description of "Figure 51 Connections Between Spine Switch and Leaf Switches in a Single Rack"
The Oracle Database servers and Exadata Storage Servers connect to the leaf switches as shown in the following graphic:
Figure 52 Connections Between Database Servers and Storage Servers and Leaf Switches
Description of "Figure 52 Connections Between Database Servers and Storage Servers and Leaf Switches"
When connecting up to eight racks together, remove the seven existing interswitch connections between each leaf switch, as well as the two connections between the leaf switches and the spine switch. From each leaf switch, distribute eight connections over the spine switches in all racks. In multirack environments, the leaf switches inside a rack are no longer directly interconnected, as shown in the following graphic:
Figure 53 Connections Between Spine Switches and Leaf Switches Across Two Racks
Description of "Figure 53 Connections Between Spine Switches and Leaf Switches Across Two Racks"
As shown in the preceding graphic, each leaf switch in rack 1 connects to the following switches:

Four connections to its internal spine switch

Four connections to the spine switch in rack 2
The spine switch in rack 1 connects to the following switches:

Eight connections to both internal leaf switches

Eight connections to both leaf switches in rack 2
As the number of racks increases from two to eight, the pattern continues as shown in the following graphic:
Figure 54 Connections Between Spine Switches and Leaf Switches for up to 8 Racks
Description of "Figure 54 Connections Between Spine Switches and Leaf Switches for up to 8 Racks"
As shown in the preceding graphic, each leaf switch has eight interswitch connections distributed over all spine switches. Each spine switch has 16 interswitch connections distributed over all leaf switches. The leaf switches are not directly interconnected with other leaf switches, and the spine switches are not directly interconnected with the other spine switches.
Note:
Cable lengths are specified for racks 1 through 8.
5.1.1 Preparing to Cable Racks Together
Racks can be added together to increase system capacity and performance. When cabling racks together, note the following:

Oracle recommends that the names for the servers include the rack unit number. This helps identify the server during diagnostics.

When completing Oracle Exadata Deployment Assistant for the additional rack, you are prompted for SCAN addresses. However, these SCAN addresses are not used because the SCAN address from the original rack are used. Manually remove the new SCAN addresses from the generated installation files.

The software owner account names and group names, as well as their identifiers, must match the names and identifiers of the original rack.

If the additional grid disks are used with existing disk groups, then ensure the grid disk sizes for the new rack are the same as the original rack.

If the InfiniBand network consists of four or more racks cabled together, then disable the Subnet Manager on the leaf switches.

Verify the Master Subnet Manager is located on the spine switch.

Oracle Exadata Database Machine Quarter Racks can be extended as follows:

Connect two Oracle Exadata Database Machine Quarter Racks together. At least four of the six ports reserved for external connectivity are open on each leaf switch. The six ports are 5B, 6A, 6B, 7A, 7B, and 12A in each leaf switch. Maintain the existing seven interswitch links between the leaf switches within each rack. Connect the leaf switches between the racks with two links each, using the ports reserved for external connectivity.

Connect one Oracle Exadata Database Machine Quarter Rack with one Oracle Exadata Database Machine Half Rack or one Oracle Exadata Database Machine Full Rack. At least four ports reserved for external connectivity are open on each leaf switch. The spine switch in the Oracle Exadata Database Machine Half Rack or Oracle Exadata Database Machine Full Rack remains as the spine switch. Maintain the existing seven interswitch links between the leaf switches within each rack. Connect the leaf switches between the racks with two links each, using the ports reserved for external connectivity.

Connect one Oracle Exadata Database Machine Quarter Rack with two or more Oracle Exadata Database Machine Half Racks or Oracle Exadata Database Machine Full Racks. The racks are interconnected using a fattree topology. Connect each leaf switch in the quarter rack to the spine switch of each half rack or full rack using two links each. If there are more than four racks, then use one link instead of two. The seven interswitch links between the leaf switches in the quarter rack are removed.
Note:
To connect more than one quarter rack to additional racks, it is necessary to purchase Sun Datacenter InfiniBand Switch 36 switches for the quarter racks.


If you are extending Oracle Exadata Database Machine X42 or later, or Oracle Exadata Database Machine X38 Full Rack, or Oracle Exadata Database Machine X22 (with X4170 and X4275 servers) half rack, then order the expansion kit that includes a Sun Datacenter InfiniBand Switch 36 switch.
Perform the following tasks before cabling racks together:

Determine the number of racks that will be cabled together.

Count the spare cables from the kit, and existing interswitch cables.
Note:
Oracle Exadata Database Machine X42 and later racks or Oracle Exadata Database Machine X38 Full Racks with Exadata Storage Server X42L Servers do not include spare cables or a third Sun Datacenter InfiniBand Switch 36 switch. To extend Oracle Exadata Database Machine X42 and later racks or Oracle Exadata Database Machine X38 Full Racks with Exadata Storage Server X42L Servers, you must order cables and a Sun Datacenter InfiniBand Switch 36 switch.
For Oracle Exadata Racks earlier than Oracle Exadata Database Machine X42, no additional InfiniBand cables need to be purchased when connecting up to three Oracle Exadata Database Machine Full Racks. The following table lists the spare cables for the switch:
Cable Spare Interswitch Total InfiniBand 2 meter cable
0
7
7
InfiniBand 3 meter cable
6
2
8
InfiniBand 5 meter cable
10
0
10
When connecting four or more racks, it is necessary to purchase additional 10 meter optical QSFPtoQSFP QDR InfiniBand cables. Two QSFP SR transceivers are needed for each optical cable. The following table lists the number of racks, and cables needed to interconnect the racks.
Number of Racks 3 Meter Cables Needed 5 Meter Cables Needed 10 Meter Cables Needed 2
16
16
0
3
18
30
0
4
16
40
8
5
20
42
18
6
24
46
26
7
28
44
40
8
16
52
60

Gather the following tools:

Pen

Note pad

Flashlight

Labels for cables and rack

Cable ties (Oracle recommends hook and loop straps)
See Also:
Oracle Exadata Database Machine Maintenance Guide for information about cable labels


Determine a naming method for the rack prefixes. For example, if the original rack has the prefix
dm01
, then use the prefixdm02
for the second rack, the prefixdm03
for the third rack, and so on. 
Verify the racks have unique host names and IP addresses. All servers interconnected in the racks must have unique names and IP addresses.
Server names and IP addresses conventions may differ in the following cases:

Initial installation of all Oracle Exadata Database Machine Full Racks: System address assignments and host names should be complete.

New Oracle Exadata Database Machine Full Racks are added to an existing cluster: The new rack configuration should require unique host names and IP addresses for the new Oracle Exadata Database Machines. The IP addresses on the same subnet cannot conflict with the existing systems.

Two existing Oracle Exadata Database Machine Full Racks are clustered together: You can assign host names and IP addresses only if Oracle Exadata Database Machines are already assigned unique host names and IP addresses, or the entire cluster must be reconfigured. The machines must be on the same subnet and not have conflicting IP addresses.


Ensure the IP addresses for the new servers are in the same subnet, and do not overlap with the currentlyinstalled servers.

Ensure the firmware on the original switches are at the same level as the new switches using the
nm2version
command. If the firmware is not at the same level, then apply a firmware patch.
5.1.2 Cabling Oracle Exadata Quarter Racks and Oracle Exadata Eighth Racks
Oracle Exadata Quarter Racks and Oracle Exadata Eighth Racks can be cabled as follows:

Oracle Exadata Quarter Rack to Oracle Exadata Quarter Rack

Oracle Exadata Quarter Rack to Oracle Exadata Half Rack, or multiple Oracle Exadata Half Racks or Oracle Exadata Full Racks

Oracle Exadata Quarter Rack to Oracle Exadata Full Rack, or multiple Oracle Exadata Full Racks or Oracle Exadata Half Racks

Oracle Exadata Eighth Rack to Oracle Exadata Eighth Rack

Oracle Exadata Eighth Rack to Oracle Exadata Half Rack, or multiple Oracle Exadata Half Racks or Oracle Exadata Full Racks

Oracle Exadata Eighth Rack to Oracle Exadata Full Rack, or multiple Oracle Exadata Full Racks or Oracle Exadata Half Racks
Note:

To cable additional Oracle Exadata Quarter Racks or Oracle Exadata Eighth Racks, it is necessary to purchase additional Sun Datacenter InfiniBand Switch 36 switches.

It is assumed that four of the six ports dedicated for external connectivity in Oracle Exadata Half Rack or Oracle Exadata Full Rack are unused and available for switch interconnections with Oracle Exadata Quarter Rack or Oracle Exadata Eighth Rack. The six ports are 5B, 6A, 6B, 7A, 7B, and 12A in each leaf switch.
The following graphic shows the cable connections for two Oracle Exadata Quarter Racks. The leaf switches within each rack maintain their existing seven connections. The leaf switches interconnect between the racks with two links each using the ports reserved for external connectivity.
Figure 55 Leaf Switch Connections Between 2 Quarter Racks
Description of "Figure 55 Leaf Switch Connections Between 2 Quarter Racks"
The following graphic shows the cable connections from Oracle Exadata Quarter Rack to Oracle Exadata Half Rack or Oracle Exadata Full Rack. The leaf switches within each rack maintain their existing seven connections. The leaf switches interconnect between the racks with two links each using the ports reserved for external connectivity.
Figure 56 Leaf and Spine Switch Connections Between a Quarter Rack and a Half or Full Rack
Description of "Figure 56 Leaf and Spine Switch Connections Between a Quarter Rack and a Half or Full Rack"
The following graphic shows the cable connections from Oracle Exadata Quarter Rack to two or more racks. The racks that connect to Oracle Exadata Quarter Rack must be all Oracle Exadata Half Racks or Oracle Exadata Full Racks, interconnected using a fattree topology. Each leaf switch in Oracle Exadata Quarter Rack connects to the spine switches in the other half racks or full racks with two links each. If there are more than four racks, then use one link instead of two. The seven interswitch links between the leaf switches in the quarter rack are removed.
Figure 57 Leaf and Spine Switch Connections for a Quarter Rack Connected to One or More Half or Full Racks
Description of "Figure 57 Leaf and Spine Switch Connections for a Quarter Rack Connected to One or More Half or Full Racks"
5.2 TwoRack Cabling
Review this information before cabling two racks together.

The spine switch is in U1 (also referred to as IB1) for all racks.

In the following lists, the leaf switches are referred to as IB2 and IB3. Use the rack unit that is appropriate for your system.

In Oracle Exadata Database Machine X22 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers, the leaf switches are in U20 and U24, also referred to as IB2 and IB3, respectively.

In Oracle Exadata Database Machine 8socket (X88, X78, X68, X58, X48, X38 and X28) Full Rack systems, the leaf switches are in U21 and U23, also referred to as IB2 and IB3, respectively.

In Oracle Exadata Database Machine X42 and later racks, Oracle Exadata Database Machine X32 Full Racks and Oracle Exadata Storage Expansion Rack X32 racks, the leaf switches are in U20 and U22, also referred to as IB2 and IB3, respectively.

The cable lengths shown in the following lists assume that the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cables lengths. Cables lengths up to 100 meters are supported.
The following table shows the cable connections for the first spine switch (R1U1) when cabling two full racks together.
Table 51 Leaf Switch Connections for the First Rack in a TwoRack System
Leaf Switch  Connection  Cable Length 

R1 IB3 within Rack 1 
R1IB3P8A to R1U1P3A 
3 meters 
R1 IB3 to Rack 2 
R1IB3P10A to R2U1P7A 
5 meters 
R1 IB2 within Rack 1 
R1IB2P8A to R1U1P3B 
3 meters 
R1 IB2 to Rack 2 
R1IB2P10A to R2U1P7B 
5 meters 
The following table shows the cable connections for the second spine switch (R2U1) when cabling two full racks together.
Table 52 Leaf Switch Connections for the Second Rack in a TwoRack System
Leaf Switch  Connection  Cable Length 

R2 IB3 within Rack 2 
R2IB3P8A to R2U1P3A 
3 meters 
R2 IB3 to Rack 1 
R2IB3P10A to R1U1P7A 
5 meters 
R2 IB2 within Rack 2 
R2IB2P8A to R2U1P3B 
3 meters 
R2 IB2 to Rack 1 
R2IB2P10A to R1U1P7B 
5 meters 
Related Topics
5.3 ThreeRack Cabling
Note the following when cabling three racks together:

The spine switch is in U1 (also referred to as IB1) for all racks.

In the following tables, the leaf switches are referred to as IB2 and IB3. Use the rack unit that is appropriate for your system.

In Oracle Exadata Database Machine X22 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers, the leaf switches are in U20 and U24, also referred to as IB2 and IB3, respectively.

In Oracle Exadata Database Machine X28 and later, the leaf switches are in U21 and U23, also referred to as IB2 and IB3, respectively.

In Oracle Exadata Database Machine X42 and later racks, or Oracle Exadata Database Machine X32 Full Racks or Oracle Exadata Storage Expansion Rack X32 racks, the leaf switches are in U20 and U22, also referred to as IB2 and IB3, respectively.

The cable lengths shown in the tables assume the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cables lengths. Up to 100 meters is supported.

Only optical cables are supported for lengths greater than 5 meters.
Table 53 shows the cable connections for the first spine switch (R1U1) when cabling three racks together:
Table 53 Leaf Switch Connections for the First Rack in a ThreeRack System
Leaf Switch  Connection  Cable Length 

R1 IB3 within Rack 1 
R1IB3P8A to R1U1P3A R1IB3P8B to R1U1P4A R1IB3P9A to R1U1P5A 
3 meters 
R1 IB3 to Rack 2 
R1IB3P9B to R2U1P6A R1IB3P10A to R2U1P7A R1IB3P10B to R2U1P8A 
5 meters 
R1 IB3 to Rack 3 
R1IB3P11A to R3U1P9A R1IB3P11B to R3U1P10A 
5 meters 
R1 IB2 within Rack 1 
R1IB2P8A to R1U1P3B R1IB2P8B to R1U1P4B R1IB2P9A to R1U1P5B 
3 meters 
R1 IB2 to Rack 2 
R1IB2P9B to R2U1P6B R1IB2P10A to R2U1P7B R1IB2P10B to R2U1P8B 
5 meters 
R1 IB2 to Rack 3 
R1IB2P11A to R3U1P9B R1IB2P11B to R3U1P10B 
5 meters 
Table 54 shows the cable connections for the second spine switch (R2U1) when cabling three racks together:
Table 54 Leaf Switch Connections for the Second Rack in a ThreeRack System
Leaf Switch  Connection  Cable Length 

R2 IB3 within Rack 2 
R2IB3P8A to R2U1P3A R2IB3P8B to R2U1P4A R2IB3P9A to R2U1P5A 
3 meters 
R2 IB3 to Rack 1 
R2IB3P11A to R1U1P9A R2IB3P11B to R1U1P10A 
5 meters 
R2 IB3 to Rack 3 
R2IB3P9B to R3U1P6A R2IB3P10A to R3U1P7A R2IB3P10B to R3U1P8A 
5 meters 
R2 IB2 within Rack 2 
R2IB2P8A to R2U1P3B R2IB2P8B to R2U1P4B R2IB2P9A to R2U1P5B 
3 meters 
R2 IB2 to Rack 1 
R2IB2P11A to R1U1P9B R2IB2P11B to R1U1P10B 
5 meters 
R2 IB2 to Rack 3 
R2IB2P9B to R3U1P6B R2IB2P10A to R3U1P7B R2IB2P10B to R3U1P8B 
5 meters 
Table 55 shows the cable connections for the third spine switch (R3U1) when cabling three full racks together:
Table 55 Leaf Switch Connections for the Third Rack in a ThreeRack System
Leaf Switch  Connection  Cable Length 

R3 IB3 within Rack 3 
R3IB3P8A to R3U1P3A R3IB3P8B to R3U1P4A R3IB3P9A to R3U1P5A 
3 meters 
R3 IB3 to Rack 1 
R3IB3P9B to R1U1P6A R3IB3P10A to R1U1P7A R3IB3P10B to R1U1P8A 
5 meters 
R3 IB3 to Rack 2 
R3IB3P11A to R2U1P9A R3IB3P11B to R2U1P10A 
5 meters 
R3 IB2 within Rack 3 
R3IB2P8A to R3U1P3B R3IB2P8B to R3U1P4B R3IB2P9A to R3U1P5B 
3 meters 
R3 IB2 to Rack 1 
R3IB2P9B to R1U1P6B R3IB2P10A to R1U1P7B R3IB2P10B to R1U1P8B 
5 meters 
R3 IB2 to Rack 2 
R3IB2P11A to R2U1P9B R3IB2P11B to R2U1P10B 
5 meters 
5.4 FourRack Cabling
Note the following when cabling four racks together:

The spine switch is in U1 (also referred to as IB1) for all racks.

In the following tables, the leaf switches are referred to as IB2 and IB3. Use the rack unit that is appropriate for your system.

In Oracle Exadata Database Machine X22 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers, the leaf switches are in U20 and U24, also referred to as IB2 and IB3, respectively.

In Oracle Exadata Database Machine X28 and later, the leaf switches are in U21 and U23, also referred to as IB2 and IB3, respectively.

In Oracle Exadata Database Machine X42 and later racks, or Oracle Exadata Database Machine X32 Full Racks or Oracle Exadata Storage Expansion Rack X32 racks, the leaf switches are in U20 and U22, also referred to as IB2 and IB3, respectively.

The cable lengths shown in the tables assume the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cables lengths. Up to 100 meters is supported.

Only optical cables are supported for lengths greater than 5 meters.
Table 56 shows the cable connections for the first spine switch (R1U1) when cabling four racks together:
Table 56 Leaf Switch Connections for the First Rack in a FourRack System
Leaf Switch  Connection  Cable Length 

R1 IB3 within Rack 1 
R1IB3P8A to R1U1P3A R1IB3P8B to R1U1P4A 
3 meters 
R1 IB3 to Rack 2 
R1IB3P9A to R2U1P5A R1IB3P9B to R2U1P6A 
5 meters 
R1 IB3 to Rack 3 
R1IB3P10A to R3U1P7A R1IB3P10B to R3U1P8A 
5 meters 
R1 IB3 to Rack 4 
R1IB3P11A to R4U1P9A R1IB3P11B to R4U1P10A 
10 meters 
R1 IB2 within Rack 1 
R1IB2P8A to R1U1P3B R1IB2P8B to R1U1P4B 
3 meters 
R1 IB2 to Rack 2 
R1IB2P9A to R2U1P5B R1IB2P9B to R2U1P6B 
5 meters 
R1 IB2 to Rack 3 
R1IB2P10A to R3U1P7B R1IB2P10B to R3U1P8B 
5 meters 
R1 IB2 to Rack 4 
R1IB2P11A to R4U1P9B R1IB2P11B to R4U1P10B 
10 meters 
Table 57 shows the cable connections for the second spine switch (R2U1) when cabling four full racks together:
Table 57 Leaf Switch Connections for the Second Rack in a FourRack System
Leaf Switch  Connection  Cable Length 

R2 IB3 within Rack 2 
R2IB3P8A to R2U1P3A R2IB3P8B to R2U1P4A 
3 meters 
R2 IB3 to Rack 1 
R2IB3P11A to R1U1P9A R2IB3P11B to R1U1P10A 
5 meters 
R2 IB3 to Rack 3 
R2IB3P9A to R3U1P5A R2IB3P9B to R3U1P6A 
5 meters 
R2 IB3 to Rack 4 
R2IB3P10A to R4U1P7A R2IB3P10B to R4U1P8A 
5 meters 
R2 IB2 within Rack 2 
R2IB2P8A to R2U1P3B R2IB2P8B to R2U1P4B 
3 meters 
R2 IB2 to Rack 1 
R2IB2P11A to R1U1P9B R2IB2P11B to R1U1P10B 
5 meters 
R2 IB2 to Rack 3 
R2IB2P9A to R3U1P5B R2IB2P9B to R3U1P6B 
5 meters 
R2 IB2 to Rack 4 
R2IB2P10A to R4U1P7B R2IB2P10B to R4U1P8B 
5 meters 
Table 58 shows the cable connections for the third spine switch (R3U1) when cabling four full racks together:
Table 58 Leaf Switch Connections for the Third Rack in a FourRack System
Leaf Switch  Connection  Cable Length 

R3 IB3 within Rack 3 
R3IB3P8A to R3U1P3A R3IB3P8B to R3U1P4A 
3 meters 
R3 IB3 to Rack 1 
R3IB3P10A to R1U1P7A R3IB3P10B to R1U1P8A 
5 meters 
R3 IB3 to Rack 2 
R3IB3P11A to R2U1P9A R3IB3P11B to R2U1P10A 
5 meters 
R3 IB3 to Rack 4 
R3IB3P9A to R4U1P5A R3IB3P9B to R4U1P6A 
5 meters 
R3 IB2 within Rack 3 
R3IB2P8A to R3U1P3B R3IB2P8B to R3U1P4B 
3 meters 
R3 IB2 to Rack 1 
R3IB2P10A to R1U1P7B R3IB2P10B to R1U1P8B 
5 meters 
R3 IB2 to Rack 2 
R3IB2P11A to R2U1P9B R3IB2P11B to R2U1P10B 
5 meters 
R3 IB2 to Rack 4 
R3IB2P9A to R4U1P5B R3IB2P9B to R4U1P6B 
5 meters 
Table 59 shows the cable connections for the fourth spine switch (R4U1) when cabling four full racks together:
Table 59 Leaf Switch Connections for the Fourth Rack in a FourRack System
Leaf Switch  Connection  Cable Length 

R4 IB3 within Rack 4 
R4IB3P8A to R4U1P3A R4IB3P8B to R4U1P4A 
3 meters 
R4 IB3 to Rack 1 
R4IB3P9A to R1U1P5A R4IB3P9B to R1U1P6A 
10 meters 
R4 IB3 to Rack 2 
R4IB3P10A to R2U1P7A R4IB3P10B to R2U1P8A 
5 meters 
R4 IB3 to Rack 3 
R4IB3P11A to R3U1P9A R4IB3P11B to R3U1P10A 
5 meters 
R4 IB2 within Rack 4 
R4IB2P8A to R4U1P3B R4IB2P8B to R4U1P4B 
3 meters 
R4 IB2 to Rack 1 
R4IB2P9A to R1U1P5B R4IB2P9B to R1U1P6B 
10 meters 
R4 IB2 to Rack 2 
R4IB2P10A to R2U1P7B R4IB2P10B to R2U1P8B 
5 meters 
R4 IB2 to Rack 3 
R4IB2P11A to R3U1P9B R4IB2P11B to R3U1P10B 
5 meters 
5.5 FiveRack Cabling
Note the following when cabling five racks together:

The spine switch is in U1 (also referred to as IB1) for all racks.

In the following tables, the leaf switches are referred to as IB2 and IB3. Use the rack unit that is appropriate for your system.

In Oracle Exadata Database Machine X22 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers, the leaf switches are in U20 and U24, also referred to as IB2 and IB3, respectively.

In Oracle Exadata Database Machine X28 and later racks, the leaf switches are in U21 and U23, also referred to as IB2 and IB3, respectively.

In Oracle Exadata Database Machine X42 and later racks, or Oracle Exadata Database Machine X32 Full Racks and Oracle Exadata Storage Expansion Rack X32 racks, the leaf switches are in U20 and U22, also referred to as IB2 and IB3, respectively.

The cable lengths shown in the tables assume the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cables lengths. Up to 100 meters is supported.

Only optical cables are supported for lengths greater than 5 meters.
Table 510 shows the cable connections for the first spine switch (R1U1) when cabling five racks together:
Table 510 Leaf Switch Connections for the First Rack in a FiveRack System
Leaf Switch  Connection  Cable Length 

R1 IB3 within Rack 1 
R1IB3P8A to R1U1P3A R1IB3P8B to R1U1P4A 
3 meters 
R1 IB3 to Rack 2 
R1IB3P9A to R2U1P5A R1IB3P9B to R2U1P6A 
5 meters 
R1 IB3 to Rack 3 
R1IB3P10A to R3U1P7A R1IB3P10B to R3U1P8A 
5 meters 
R1 IB3 to Rack 4 
R1IB3P11A to R4U1P9A 
10 meters 
R1 IB3 to Rack 5 
R1IB3P11B to R5U1P10A 
10 meters 
R1 IB2 within Rack 1 
R1IB2P8A to R1U1P3B R1IB2P8B to R1U1P4B 
3 meters 
R1 IB2 to Rack 2 
R1IB2P9A to R2U1P5B R1IB2P9B to R2U1P6B 
5 meters 
R1 IB2 to Rack 3 
R1IB2P10A to R3U1P7B R1IB2P10B to R3U1P8B 
5 meters 
R1 IB2 to Rack 4 
R1IB2P11A to R4U1P9B 
10 meters 
R1 IB2 to Rack 5 
R1IB2P11B to R5U1P10B 
10 meters 
Table 511 shows the cable connections for the second spine switch (R2U1) when cabling five full racks together:
Table 511 Leaf Switch Connections for the Second Rack in a FiveRack System
Leaf Switch  Connection  Cable Length 

R2 IB3 within Rack 2 
R2IB3P8A to R2U1P3A R2IB3P8B to R2U1P4A 
3 meters 
R2 IB3 to Rack 1 
R2IB3P11B to R1U1P10A 
5 meters 
R2 IB3 to Rack 3 
R2IB3P9A to R3U1P5A R2IB3P9B to R3U1P6A 
5 meters 
R2 IB3 to Rack 4 
R2IB3P10A to R4U1P7A R2IB3P10B to R4U1P8A 
5 meters 
R2 IB3 to Rack 5 
R2IB3P11A to R5U1P9A 
10 meters 
R2 IB2 within Rack 2 
R2IB2P8A to R2U1P3B R2IB2P8B to R2U1P4B 
3 meters 
R2 IB2 to Rack 1 
R2IB2P11B to R1U1P10B 
5 meters 
R2 IB2 to Rack 3 
R2IB2P9A to R3U1P5B R2IB2P9B to R3U1P6B 
5 meters 
R2 IB2 to Rack 4 
R2IB2P10A to R4U1P7B R2IB2P10B to R4U1P8B 
5 meters 
R2 IB2 to Rack 5 
R2IB2P11A to R5U1P9B 
10 meters 
Table 512 shows the cable connections for the third spine switch (R3U1) when cabling five full racks together:
Table 512 Leaf Switch Connections for the Third Rack in a FiveRack System
Leaf Switch  Connection  Cable Length 

R3 IB3 within Rack 3 
R3IB3P8A to R3U1P3A R3IB3P8B to R3U1P4A 
3 meters 
R3 IB3 to Rack 1 
R3IB3P11A to R1U1P9A 
5 meters 
R3 IB3 to Rack 2 
R3IB3P11B to R2U1P10A 
5 meters 
R3 IB3 to Rack 4 
R3IB3P9A to R4U1P5A R3IB3P9B to R4U1P6A 
5 meters 
R3 IB3 to Rack 5 
R3IB3P10A to R5U1P7A R3IB3P10B to R5U1P8A 
5 meters 
R3 IB2 within Rack 3 
R3IB2P8A to R3U1P3B R3IB2P8B to R3U1P4B 
3 meters 
R3 IB2 to Rack 1 
R3IB2P11A to R1U1P9B 
5 meters 
R3 IB2 to Rack 2 
R3IB2P11B to R2U1P10B 
5 meters 
R3 IB2 to Rack 4 
R3IB2P9A to R4U1P5B R3IB2P9B to R4U1P6B 
5 meters 
R3 IB2 to Rack 5 
R3IB2P10A to R5U1P7B R3IB2P10B to R5U1P8B 
5 meters 
Table 513 shows the cable connections for the fourth spine switch (R4U1) when cabling five full racks together:
Table 513 Leaf Switch Connections for the Fourth Rack in a FiveRack System
Leaf Switch  Connection  Cable Length 

R4 IB3 within Rack 4 
R4IB3P8A to R4U1P3A R4IB3P8B to R4U1P4A 
3 meters 
R4 IB3 to Rack 1 
R4IB3P10A to R1U1P7A R4IB3P10B to R1U1P8A 
10 meters 
R4 IB3 to Rack 2 
R4IB3P11A to R2U1P9A 
5 meters 
R4 IB3 to Rack 3 
R4IB3P11B to R3U1P10A 
5 meters 
R4 IB3 to Rack 5 
R4IB3P9A to R5U1P5A R4IB3P9B to R5U1P6A 
5 meters 
R4 IB2 within Rack 4 
R4IB2P8A to R4U1P3B R4IB2P8B to R4U1P4B 
3 meters 
R4 IB2 to Rack 1 
R4IB2P10A to R1U1P7B R4IB2P10B to R1U1P8B 
10 meters 
R4 IB2 to Rack 2 
R4IB2P11A to R2U1P9B 
5 meters 
R4 IB2 to Rack 3 
R4IB2P11B to R3U1P10B 
5 meters 
R4 IB2 to Rack 5 
R4IB2P9A to R5U1P5B R4IB2P9B to R5U1P6B 
5 meters 
Table 514 shows the cable connections for the fifth spine switch (R5U1) when cabling five full racks together:
Table 514 Leaf Switch Connections for the Fifth Rack in a FiveRack System
Leaf Switch  Connection  Cable Length 

R5 IB3 within Rack 5 
R5IB3P8A to R5U1P3A R5IB3P8B to R5U1P4A 
3 meters 
R5 IB3 to Rack 1 
R5IB3P9A to R1U1P5A R5IB3P9B to R1U1P6A 
10 meters 
R5 IB3 to Rack 2 
R5IB3P10A to R2U1P7A R5IB3P10B to R2U1P8A 
10 meters 
R5 IB3 to Rack 3 
R5IB3P11A to R3U1P9A 
5 meters 
R5 IB3 to Rack 4 
R5IB3P11B to R4U1P10A 
5 meters 
R5 IB2 within Rack 5 
R5IB2P8A to R5U1P3B R5IB2P8B to R5U1P4B 
3 meters 
R5 IB2 to Rack 1 
R5IB2P9A to R1U1P5B R5IB2P9B to R1U1P6B 
10 meters 
R5 IB2 to Rack 2 
R5IB2P10A to R2U1P7B R5IB2P10B to R2U1P8B 
10 meters 
R5 IB2 to Rack 3 
R5IB2P11A to R3U1P9B 
5 meters 
R5 IB2 to Rack 4 
R5IB2P11B to R4U1P10B 
5 meters 
5.6 SixRack Cabling
Note the following when cabling six racks together:

The spine switch is in U1 (also referred to as IB1) for all racks.

In the following tables, the leaf switches are referred to as IB2 and IB3. Use the rack unit that is appropriate for your system.

In Oracle Exadata Database Machine X22 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers, the leaf switches are in U20 and U24, also referred to as IB2 and IB3, respectively.

In Oracle Exadata Database Machine X28 and later 8socket racks, the leaf switches are in U21 and U23, also referred to as IB2 and IB3, respectively.

In Oracle Exadata Database Machine X42 and later twosocket racks, or Oracle Exadata Database Machine X32 Full Racks and Oracle Exadata Storage Expansion Rack X32 racks, the leaf switches are in U20 and U22, also referred to as IB2 and IB3, respectively.

The cable lengths shown in the tables assume the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cables lengths. Up to 100 meters is supported.

Only optical cables are supported for lengths greater than 5 meters.
Table 515 shows the cable connections for the first spine switch (R1U1) when cabling six racks together:
Table 515 Leaf Switch Connections for the First Rack in a SixRack System
Leaf Switch  Connection  Cable Length 

R1 IB3 within Rack 1 
R1IB3P8A to R1U1P3A R1IB3P8B to R1U1P4A 
3 meters 
R1 IB3 to Rack 2 
R1IB3P9A to R2U1P5A R1IB3P9B to R2U1P6A 
5 meters 
R1 IB3 to Rack 3 
R1IB3P10A to R3U1P7A 
5 meters 
R1 IB3 to Rack 4 
R1IB3P10B to R4U1P8A 
10 meters 
R1 IB3 to Rack 5 
R1IB3P11A to R5U1P9A 
10 meters 
R1 IB3 to Rack 6 
R1IB3P11B to R6U1P10A 
10 meters 
R1 IB2 within Rack 1 
R1IB2P8A to R1U1P3B R1IB2P8B to R1U1P4B 
3 meters 
R1 IB2 to Rack 2 
R1IB2P9A to R2U1P5B R1IB2P9B to R2U1P6B 
5 meters 
R1 IB2 to Rack 3 
R1IB2P10A to R3U1P7B 
5 meters 
R1 IB2 to Rack 4 
R1IB2P10B to R4U1P8B 
10 meters 
R1 IB2 to Rack 5 
R1IB2P11A to R5U1P9B 
10 meters 
R1 IB2 to Rack 6 
R1IB2P11B to R6U1P10B 
10 meters 
Table 516 shows the cable connections for the second spine switch (R2U1) when cabling six full racks together:
Table 516 Leaf Switch Connections for the Second Rack in a SixRack System
Leaf Switch  Connection  Cable Length 

R2 IB3 within Rack 2 
R2IB3P8A to R2U1P3A R2IB3P8B to R2U1P4A 
3 meters 
R2 IB3 to Rack 1 
R2IB3P11B to R1U1P10A 
5 meters 
R2 IB3 to Rack 3 
R2IB3P9A to R3U1P5A R2IB3P9B to R3U1P6A 
5 meters 
R2 IB3 to Rack 4 
R2IB3P10A to R4U1P7A 
5 meters 
R2 IB3 to Rack 5 
R2IB3P10B to R5U1P8A 
10 meters 
R2 IB3 to Rack 6 
R2IB3P11Ato R6U1P9A 
10 meters 
R2 IB2 within Rack 2 
R2IB2P8A to R2U1P3B R2IB2P8B to R2U1P4B 
3 meters 
R2 IB2 to Rack 1 
R2IB2P11B to R1U1P10B 
5 meters 
R2 IB2 to Rack 3 
R2IB2P9A to R3U1P5B R2IB2P9B to R3U1P6B 
5 meters 
R2 IB2 to Rack 4 
R2IB2P10A to R4U1P7B 
5 meters 
R2 IB2 to Rack 5 
R2IB2P10B to R5U1P8B 
10 meters 
R2 IB2 to Rack 6 
R2IB2P11Ato R6U1P9B 
10 meters 
Table 517 shows the cable connections for the third spine switch (R3U1) when cabling six full racks together:
Table 517 Leaf Switch Connections for the Third Rack in a SixRack System
Leaf Switch  Connection  Cable Length 

R3 IB3 within Rack 3 
R3IB3P8A to R3U1P3A R3IB3P8B to R3U1P4A 
3 meters 
R3 IB3 to Rack 1 
R3IB3P11A to R1U1P9A 
5 meters 
R3 IB3 to Rack 2 
R3IB3P11B to R2U1P10A 
5 meters 
R3 IB3 to Rack 4 
R3IB3P9A to R4U1P5A R3IB3P9B to R4U1P6A 
5 meters 
R3 IB3 to Rack 5 
R3IB3P10A to R5U1P7A 
5 meters 
R3 IB3 to Rack 6 
R3IB3P10B to R6U1P8A 
10 meters 
R3 IB2 within Rack 3 
R3IB2P8A to R3U1P3B R3IB2P8B to R3U1P4B 
3 meters 
R3 IB2 to Rack 1 
R3IB2P11A to R1U1P9B 
5 meters 
R3 IB2 to Rack 2 
R3IB2P11B to R2U1P10B 
5 meters 
R3 IB2 to Rack 4 
R3IB2P9A to R4U1P5B R3IB2P9B to R4U1P6B 
5 meters 
R3 IB2 to Rack 5 
R3IB2P10A to R5U1P7B 
5 meters 
R3 IB2 to Rack 6 
R3IB2P10B to R6U1P8B 
10 meters 
Table 518 shows the cable connections for the fourth spine switch (R4U1) when cabling six full racks together:
Table 518 Leaf Switch Connections for the Fourth Rack in a SixRack System
Leaf Switch  Connection  Cable Length 

R4 IB3 within Rack 4 
R4IB3P8A to R4U1P3A R4IB3P8B to R4U1P4A 
3 meters 
R4 IB3 to Rack 1 
R4IB3P10B to R1U1P8A 
10 meters 
R4 IB3 to Rack 2 
R4IB3P11A to R2U1P9A 
5 meters 
R4 IB3 to Rack 3 
R4IB3P11B to R3U1P10A 
5 meters 
R4 IB3 to Rack 5 
R4IB3P9A to R5U1P5A R4IB3P9B to R5U1P6A 
5 meters 
R4 IB3 to Rack 6 
R4IB3P10A to R6U1P7A 
5 meters 
R4 IB2 within Rack 4 
R4IB2P8A to R4U1P3B R4IB2P8B to R4U1P4B 
3 meters 
R4 IB2 to Rack 1 
R4IB2P10B to R1U1P8B 
10 meters 
R4 IB2 to Rack 2 
R4IB2P11A to R2U1P9B 
5 meters 
R4 IB2 to Rack 3 
R4IB2P11B to R3U1P10B 
5 meters 
R4 IB2 to Rack 5 
R4IB2P9A to R5U1P5B R4IB2P9B to R5U1P6B 
5 meters 
R4 IB2 to Rack 6 
R4IB2P10A to R6U1P7B 
5 meters 
Table 519 shows the cable connections for the fifth spine switch (R5U1) when cabling six full racks together:
Table 519 Leaf Switch Connections for the Fifth Rack in a SixRack System
Leaf Switch  Connection  Cable Length 

R5 IB3 within Rack 5 
R5IB3P8A to R5U1P3A R5IB3P8B to R5U1P4A 
3 meters 
R5 IB3 to Rack 1 
R5IB3P10A to R1U1P7A 
10 meters 
R5 IB3 to Rack 2 
R5IB3P10B to R2U1P8A 
10 meters 
R5 IB3 to Rack 3 
R5IB3P11A to R3U1P9A 
5 meters 
R5 IB3 to Rack 4 
R5IB3P11B to R4U1P10A 
5 meters 
R5 IB3 to Rack 6 
R5IB3P9A to R6U1P5A R5IB3P9B to R6U1P6A 
5 meters 
R5 IB2 within Rack 5 
R5IB2P8A to R5U1P3B R5IB2P8B to R5U1P4B 
3 meters 
R5 IB2 to Rack 1 
R5IB2P10A to R1U1P7B 
10 meters 
R5 IB2 to Rack 2 
R5IB2P10B to R2U1P8B 
10 meters 
R5 IB2 to Rack 3 
R5IB2P11A to R3U1P9B 
5 meters 
R5 IB2 to Rack 4 
R5IB2P11B to R4U1P10B 
5 meters 
R5 IB2 to Rack 6 
R5IB2P9A to R6U1P5B R5IB2P9B to R6U1P6B 
5 meters 
Table 520 shows the cable connections for the sixth spine switch (R6U1) when cabling six full racks together:
Table 520 Leaf Switch Connections for the Sixth Rack in a SixRack System
Leaf Switch  Connection  Cable Length 

R6 IB3 within Rack 6 
R6IB3P8A to R6U1P3A R6IB3P8B to R6U1P4A 
3 meters 
R6 IB3 to Rack 1 
R6IB3P9A to R1U1P5A R6IB3P9B to R1U1P6A 
10 meters 
R6 IB3 to Rack 2 
R6IB3P10A to R2U1P7A 
10 meters 
R6 IB3 to Rack 3 
R6IB3P10B to R3U1P8A 
10 meters 
R6 IB3 to Rack 4 
R6IB3P11A to R4U1P9A 
5 meters 
R6 IB3 to Rack 5 
R6IB3P11B to R5U1P10A 
5 meters 
R6 IB2 within Rack 6 
R6IB2P8A to R6U1P3B R6IB2P8B to R6U1P4B 
3 meters 
R6 IB2 to Rack 1 
R6IB2P9A to R1U1P5B R6IB2P9B to R1U1P6B 
10 meters 
R6 IB2 to Rack 2 
R6IB2P10A to R2U1P7B 
10 meters 
R6 IB2 to Rack 3 
R6IB2P10B to R3U1P8B 
10 meters 
R6 IB2 to Rack 4 
R6IB2P11A to R4U1P9B 
5 meters 
R6 IB2 to Rack 5 
R6IB2P11B to R5U1P10B 
5 meters 
5.7 SevenRack Cabling
Note the following when cabling five racks together:

The spine switch is in U1 (also referred to as IB1) for all racks.

In the following tables, the leaf switches are referred to as IB2 and IB3. Use the rack unit that is appropriate for your system.

In Oracle Exadata Database Machine X22 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers, the leaf switches are in U20 and U24, also referred to as IB2 and IB3, respectively.

In Oracle Exadata Database Machine X28 and later racks, the leaf switches are in U21 and U23, also referred to as IB2 and IB3, respectively.

In Oracle Exadata Database Machine X42 and later racks, or Oracle Exadata Database Machine X32 Full Racks and Oracle Exadata Storage Expansion Rack X32 racks, the leaf switches are in U20 and U22, also referred to as IB2 and IB3, respectively.

The cable lengths shown in the tables assume the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cables lengths. Up to 100 meters is supported.

Only optical cables are supported for lengths greater than 5 meters.
Table 521 shows the cable connections for the first spine switch (R1U1) when cabling seven racks together:
Table 521 Leaf Switch Connections for the First Rack in a SevenRack System
Leaf Switch  Connection  Cable Length 

R1 IB3 within Rack 1 
R1IB3P8A to R1U1P3A R1IB3P8B to R1U1P4A 
3 meters 
R1 IB3 to Rack 2 
R1IB3P9A to R2U1P5A 
5 meters 
R1 IB3 to Rack 3 
R1IB3P9B to R3U1P6A 
5 meters 
R1 IB3 to Rack 4 
R1IB3P10A to R4U1P7A 
10 meters 
R1 IB3 to Rack 5 
R1IB3P10B to R5U1P8A 
10 meters 
R1 IB3 to Rack 6 
R1IB3P11A to R6U1P9A 
10 meters 
R1 IB3 to Rack 7 
R1IB3P11B to R7U1P10A 
10 meters 
R1 IB2 within Rack 1 
R1IB2P8A to R1U1P3B R1IB2P8B to R1U1P4B 
3 meters 
R1 IB2 to Rack 2 
R1IB2P9A to R2U1P5B 
5 meters 
R1 IB2 to Rack 3 
R1IB2P9B to R3U1P6B 
5 meters 
R1 IB2 to Rack 4 
R1IB2P10A to R4U1P7B 
10 meters 
R1 IB2 to Rack 5 
R1IB2P10B to R5U1P8B 
10 meters 
R1 IB2 to Rack 6 
R1IB2P11A to R6U1P9B 
10 meters 
R1 IB2 to Rack 7 
R1IB2P11B to R7U1P10B 
10 meters 
Table 522 shows the cable connections for the second spine switch (R2U1) when cabling seven full racks together:
Table 522 Leaf Switch Connections for the Second Rack in a SevenRack System
Leaf Switch  Connection  Cable Length 

R2 IB3 within Rack 2 
R2IB3P8A to R2U1P3A R2IB3P8B to R2U1P4A 
3 meters 
R2 IB3 to Rack 1 
R2IB3P11B to R1U1P10A 
5 meters 
R2 IB3 to Rack 3 
R2IB3P9A to R3U1P5A 
5 meters 
R2 IB3 to Rack 4 
R2IB3P9B to R4U1P6A 
5 meters 
R2 IB3 to Rack 5 
R2IB3P10A to R5U1P7A 
10 meters 
R2 IB3 to Rack 6 
R2IB3P10B to R6U1P8A 
10 meters 
R2 IB3 to Rack 7 
R2IB3P11A to R7U1P9A 
10 meters 
R2 IB2 within Rack 2 
R2IB2P8A to R2U1P3B R2IB2P8B to R2U1P4B 
3 meters 
R2 IB2 to Rack 1 
R2IB2P11B to R1U1P10B 
5 meters 
R2 IB2 to Rack 3 
R2IB2P9A to R3U1P5B 
5 meters 
R2 IB2 to Rack 4 
R2IB2P9B to R4U1P6B 
5 meters 
R2 IB2 to Rack 5 
R2IB2P10A to R5U1P7B 
10 meters 
R2 IB2 to Rack 6 
R2IB2P10Bto R6U1P8B 
10 meters 
R2 IB2 to Rack 7 
R2IB2P11A to R7U1P9B 
10 meters 
Table 523 shows the cable connections for the third spine switch (R3U1) when cabling seven full racks together:
Table 523 Leaf Switch Connections for the Third Rack in a SevenRack System
Leaf Switch  Connection  Cable Length 

R3 IB3 within Rack 3 
R3IB3P8A to R3U1P3A R3IB3P8B to R3U1P4A 
3 meters 
R3 IB3 to Rack 1 
R3IB3P11A to R1U1P9A 
5 meters 
R3 IB3 to Rack 2 
R3IB3P11B to R2U1P10A 
5 meters 
R3 IB3 to Rack 4 
R3IB3P9A to R4U1P5A 
5 meters 
R3 IB3 to Rack 5 
R3IB3P9B to R5U1P6A 
5 meters 
R3 IB3 to Rack 6 
R3IB3P10A to R6U1P7A 
10 meters 
R3 IB3 to Rack 7 
R3IB3P10B to R7U1P8A 
10 meters 
R3 IB2 within Rack 3 
R3IB2P8A to R3U1P3B R3IB2P8B to R3U1P4B 
3 meters 
R3 IB2 to Rack 1 
R3IB2P11A to R1U1P9B 
5 meters 
R3 IB2 to Rack 2 
R3IB2P11B to R2U1P10B 
5 meters 
R3 IB2 to Rack 4 
R3IB2P9A to R4U1P5B 
5 meters 
R3 IB2 to Rack 5 
R3IB2P9B to R5U1P6B 
5 meters 
R3 IB2 to Rack 6 
R3IB2P10A to R6U1P7B 
10 meters 
R3 IB2 to Rack 7 
R3IB2P10B to R7U1P8B 
10 meters 
Table 524 shows the cable connections for the fourth spine switch (R4U1) when cabling seven full racks together:
Table 524 Leaf Switch Connections for the Fourth Rack in a SevenRack System
Leaf Switch  Connection  Cable Length 

R4 IB3 within Rack 4 
R4IB3P8A to R4U1P3A R4IB3P8B to R4U1P4A 
3 meters 
R4 IB3 to Rack 1 
R4IB3P10B to R1U1P8A 
10 meters 
R4 IB3 to Rack 2 
R4IB3P11A to R2U1P9A 
5 meters 
R4 IB3 to Rack 3 
R4IB3P11B to R3U1P10A 
5 meters 
R4 IB3 to Rack 5 
R4IB3P9A to R5U1P5A 
5 meters 
R4 IB3 to Rack 6 
R4IB3P9B to R6U1P6A 
5 meters 
R4 IB3 to Rack 7 
R4IB3P10A to R7U1P7A 
10 meters 
R4 IB2 within Rack 4 
R4IB2P8A to R4U1P3B R4IB2P8B to R4U1P4B 
3 meters 
R4 IB2 to Rack 1 
R4IB2P10B to R1U1P8B 
10 meters 
R4 IB2 to Rack 2 
R4IB2P11A to R2U1P9B 
5 meters 
R4 IB2 to Rack 3 
R4IB2P11B to R3U1P10B 
5 meters 
R4 IB2 to Rack 5 
R4IB2P9A to R5U1P5B 
5 meters 
R4 IB2 to Rack 6 
R4IB2P9B to R6U1P6B 
5 meters 
R4 IB2 to Rack 7 
R4IB2P10A to R7U1P7B 
10 meters 
Table 525 shows the cable connections for the fifth spine switch (R5U1) when cabling seven full racks together:
Table 525 Leaf Switch Connections for the Fifth Rack in a SevenRack System
Leaf Switch  Connection  Cable Length 

R5 IB3 within Rack 5 
R5IB3P8A to R5U1P3A R5IB3P8B to R5U1P4A 
3 meters 
R5 IB3 to Rack 1 
R5IB3P10A to R1U1P7A 
10 meters 
R5 IB3 to Rack 2 
R5IB3P10B to R2U1P8A 
10 meters 
R5 IB3 to Rack 3 
R5IB3P11A to R3U1P9A 
5 meters 
R5 IB3 to Rack 4 
R5IB3P11B to R4U1P10A 
5 meters 
R5 IB3 to Rack 6 
R5IB3P9A to R6U1P5A 
5 meters 
R5 IB3 to Rack 7 
R5IB3P9B to R7U1P6A 
5 meters 
R5 IB2 within Rack 5 
R5IB2P8A to R5U1P3B R5IB2P8B to R5U1P4B 
3 meters 
R5 IB2 to Rack 1 
R5IB2P10A to R1U1P7B 
10 meters 
R5 IB2 to Rack 2 
R5IB2P10B to R2U1P8B 
10 meters 
R5 IB2 to Rack 3 
R5IB2P11A to R3U1P9B 
5 meters 
R5 IB2 to Rack 4 
R5IB2P11B to R4U1P10B 
5 meters 
R5 IB2 to Rack 6 
R5IB2P9A to R6U1P5B 
5 meters 
R5 IB2 to Rack 7 
R5IB2P9B to R7U1P6B 
5 meters 
Table 526 shows the cable connections for the sixth spine switch (R6U1) when cabling seven full racks together:
Table 526 Leaf Switch Connections for the Sixth Rack in a SevenRack System
Leaf Switch  Connection  Cable Length 

R6 IB3 within Rack 6 
R6IB3P8A to R6U1P3A R6IB3P8B to R6U1P4A 
3 meters 
R6 IB3 to Rack 1 
R6IB3P9B to R1U1P6A 
10 meters 
R6 IB3 to Rack 2 
R6IB3P10A to R2U1P7A 
10 meters 
R6 IB3 to Rack 3 
R6IB3P10B to R3U1P8A 
10 meters 
R6 IB3 to Rack 4 
R6IB3P11A to R4U1P9A 
5 meters 
R6 IB3 to Rack 5 
R6IB3P11B to R5U1P10A 
5 meters 
R6 IB3 to Rack 7 
R6IB3P9A to R7U1P5A 
5 meters 
R6 IB2 within Rack 6 
R6IB2P8A to R6U1P3B R6IB2P8B to R6U1P4B 
3 meters 
R6 IB2 to Rack 1 
R6IB2P9B to R1U1P6B 
10 meters 
R6 IB2 to Rack 2 
R6IB2P10A to R2U1P7B 
10 meters 
R6 IB2 to Rack 3 
R6IB2P10B to R3U1P8B 
10 meters 
R6 IB2 to Rack 4 
R6IB2P11A to R4U1P9B 
5 meters 
R6 IB2 to Rack 5 
R6IB2P11B to R5U1P10B 
5 meters 
R6 IB2 to Rack 7 
R6IB2P9A to R7U1P5B 
5 meters 
Table 527 shows the cable connections for the seventh spine switch (R7U1) when cabling seven full racks together:
Table 527 Leaf Switch Connections for the Seventh Rack in a SevenRack System
Leaf Switch  Connection  Cable Length 

R7 IB3 within Rack 7 
R7IB3P8A to R7U1P3A R7IB3P8B to R7U1P4A 
3 meters 
R7 IB3 to Rack 1 
R7IB3P9A to R1U1P5A 
10 meters 
R7 IB3 to Rack 2 
R7IB3P9B to R2U1P6A 
10 meters 
R7 IB3 to Rack 3 
R7IB3P10A to R3U1P7A 
10 meters 
R7 IB3 to Rack 4 
R7IB3P10B to R4U1P8A 
10 meters 
R7 IB3 to Rack 5 
R7IB3P11A to R5U1P9A 
5 meters 
R7 IB3 to Rack 6 
R7IB3P11B to R6U1P10A 
5 meters 
R7 IB2 within Rack 7 
R7IB2P8A to R7U1P3B R7IB2P8B to R7U1P4B 
3 meters 
R7 IB2 to Rack 1 
R7IB2P9A to R1U1P5B 
10 meters 
R7 IB2 to Rack 2 
R7IB2P9B to R2U1P6B 
10 meters 
R7 IB2 to Rack 3 
R7IB2P10A to R3U1P7B 
10 meters 
R7 IB2 to Rack 4 
R7IB2P10B to R4U1P8B 
10 meters 
R7 IB2 to Rack 5 
R7IB2P11A to R5U1P9B 
5 meters 
R7 IB2 to Rack 6 
R7IB2P11B to R6U1P10B 
5 meters 
5.8 EightRack Cabling
Note the following when cabling five racks together:

The spine switch is in U1 (also referred to as IB1) for all racks.

In the following tables, the leaf switches are referred to as IB2 and IB3. Use the rack unit that is appropriate for your system.

In Oracle Exadata Database Machine X22 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers, the leaf switches are in U20 and U24, also referred to as IB2 and IB3, respectively.

In Oracle Exadata Database Machine X28 and later racks, the leaf switches are in U21 and U23, also referred to as IB2 and IB3, respectively.

In Oracle Exadata Database Machine X42 and later racks, or Oracle Exadata Database Machine X32 Full Racks and Oracle Exadata Storage Expansion Rack X32 racks, the leaf switches are in U20 and U22, also referred to as IB2 and IB3, respectively.

The cable lengths shown in the tables assume the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cables lengths. Up to 100 meters is supported.

Only optical cables are supported for lengths greater than 5 meters.
Table 528 shows the cable connections for the first spine switch (R1U1) when cabling eight racks together:
Table 528 Leaf Switch Connections for the First Rack in a EightRack System
Leaf Switch  Connection  Cable Length 

R1 IB3 within Rack 1 
R1IB3P8A to R1U1P3A 
3 meters 
R1 IB3 to Rack 2 
R1IB3P8B to R2U1P4A 
5 meters 
R1 IB3 to Rack 3 
R1IB3P9A to R3U1P5A 
5 meters 
R1 IB3 to Rack 4 
R1IB3P9B to R4U1P6A 
10 meters 
R1 IB3 to Rack 5 
R1IB3P10A to R5U1P7A 
10 meters 
R1 IB3 to Rack 6 
R1IB3P10B to R6U1P8A 
10 meters 
R1 IB3 to Rack 7 
R1IB3P11A to R7U1P9A 
10 meters 
R1 IB3 to Rack 8 
R1IB3P11B to R8U1P10A 
10 meters 
R1 IB2 within Rack 1 
R1IB2P8A to R1U1P3B 
3 meters 
R1 IB2 to Rack 2 
R1IB2P8B to R2U1P4B 
5 meters 
R1 IB2 to Rack 3 
R1IB2P9A to R3U1P5B 
5 meters 
R1 IB2 to Rack 4 
R1IB2P9B to R4U1P6B 
10 meters 
R1 IB2 to Rack 5 
R1IB2P10A to R5U1P7B 
10 meters 
R1 IB2 to Rack 6 
R1IB2P10B to R6U1P8B 
10 meters 
R1 IB2 to Rack 7 
R1IB2P11A to R7U1P9B 
10 meters 
R1 IB2 to Rack 8 
R1IB2P11B to R8U1P10B 
10 meters 
Table 529 shows the cable connections for the second spine switch (R2U1) when cabling eight full racks together:
Table 529 Leaf Switch Connections for the Second Rack in a EightRack System
Leaf Switch  Connection  Cable Length 

R2 IB3 within Rack 2 
R2IB3P8A to R2U1P3A 
3 meters 
R2 IB3 to Rack 1 
R2IB3P11B to R1U1P10A 
5 meters 
R2 IB3 to Rack 3 
R2IB3P8B to R3U1P4A 
5 meters 
R2 IB3 to Rack 4 
R2IB3P9A to R4U1P5A 
5 meters 
R2 IB3 to Rack 5 
R2IB3P9B to R5U1P6A 
10 meters 
R2 IB3 to Rack 6 
R2IB3P10A to R6U1P7A 
10 meters 
R2 IB3 to Rack 7 
R2IB3P10B to R7U1P8A 
10 meters 
R2 IB3 to Rack 8 
R2IB3P11A to R8U1P9A 
10 meters 
R2 IB2 within Rack 2 
R2IB2P8A to R2U1P3B 
3 meters 
R2 IB2 to Rack 1 
R2IB2P11B to R1U1P10B 
5 meters 
R2 IB2 to Rack 3 
R2IB2P8B to R3U1P4B 
5 meters 
R2 IB2 to Rack 4 
R2IB2P9A to R4U1P5B 
5 meters 
R2 IB2 to Rack 5 
R2IB2P9B to R5U1P6B 
10 meters 
R2 IB2 to Rack 6 
R2IB2P10A to R6U1P7B 
10 meters 
R2 IB2 to Rack 7 
R2IB2P10B to R7U1P8B 
10 meters 
R2 IB2 to Rack 8 
R2IB2P11A to R8U1P9B 
10 meters 
Table 530 shows the cable connections for the third spine switch (R3U1) when cabling eight full racks together:
Table 530 Leaf Switch Connections for the Third Rack in a EightRack System
Leaf Switch  Connection  Cable Length 

R3 IB3 within Rack 3 
R3IB3P8A to R3U1P3A 
3 meters 
R3 IB3 to Rack 1 
R3IB3P11A to R1U1P9A 
5 meters 
R3 IB3 to Rack 2 
R3IB3P11B to R2U1P10A 
5 meters 
R3 IB3 to Rack 4 
R3IB3P8B to R4U1P4A 
5 meters 
R3 IB3 to Rack 5 
R3IB3P9A to R5U1P5A 
5 meters 
R3 IB3 to Rack 6 
R3IB3P9B to R6U1P6A 
10 meters 
R3 IB3 to Rack 7 
R3IB3P10A to R7U1P7A 
10 meters 
R3 IB3 to Rack 8 
R3IB3P10B to R8U1P8A 
10 meters 
R3 IB2 within Rack 3 
R3IB2P8A to R3U1P3B 
3 meters 
R3 IB2 to Rack 1 
R3IB2P11A to R1U1P9B 
5 meters 
R3 IB2 to Rack 2 
R3IB2P11B to R2U1P10B 
5 meters 
R3 IB2 to Rack 4 
R3IB2P8B to R4U1P4B 
5 meters 
R3 IB2 to Rack 5 
R3IB2P9A to R5U1P5B 
5 meters 
R3 IB2 to Rack 6 
R3IB2P9B to R6U1P6B 
10 meters 
R3 IB2 to Rack 7 
R3IB2P10A to R7U1P7B 
10 meters 
R3 IB2 to Rack 8 
R3IB2P10B to R8U1P8B 
10 meters 
Table 531 shows the cable connections for the fourth spine switch (R4U1) when cabling eight full racks together:
Table 531 Leaf Switch Connections for the Fourth Rack in a EightRack System
Leaf Switch  Connection  Cable Length 

R4 IB3 within Rack 4 
R4IB3P8A to R4U1P3A 
3 meters 
R4 IB3 to Rack 1 
R4IB3P10B to R1U1P8A 
10 meters 
R4 IB3 to Rack 2 
R4IB3P11A to R2U1P9A 
5 meters 
R4 IB3 to Rack 3 
R4IB3P11B to R3U1P10A 
5 meters 
R4 IB3 to Rack 5 
R4IB3P8B to R5U1P4A 
5 meters 
R4 IB3 to Rack 6 
R4IB3P9A to R6U1P5A 
5 meters 
R4 IB3 to Rack 7 
R4IB3P9B to R7U1P6A 
10 meters 
R4 IB3 to Rack 8 
R4IB3P10A to R8U1P7A 
10 meters 
R4 IB2 within Rack 4 
R4IB2P8A to R4U1P3B 
3 meters 
R4 IB2 to Rack 1 
R4IB2P10B to R1U1P8B 
10 meters 
R4 IB2 to Rack 2 
R4IB2P11A to R2U1P9B 
5 meters 
R4 IB2 to Rack 3 
R4IB2P11B to R3U1P10B 
5 meters 
R4 IB2 to Rack 5 
R4IB2P8B to R5U1P4B 
5 meters 
R4 IB2 to Rack 6 
R4IB2P9A to R6U1P5B 
5 meters 
R4 IB2 to Rack 7 
R4IB2P9B to R7U1P6B 
10 meters 
R4 IB2 to Rack 8 
R4IB2P10A to R8U1P7B 
10 meters 
Table 532 shows the cable connections for the fifth spine switch (R5U1) when cabling eight full racks together:
Table 532 Leaf Switch Connections for the Fifth Rack in a EightRack System
Leaf Switch  Connection  Cable Length 

R5 IB3 within Rack 5 
R5IB3P8A to R5U1P3A 
3 meters 
R5 IB3 to Rack 1 
R5IB3P10A to R1U1P7A 
10 meters 
R5 IB3 to Rack 2 
R5IB3P10B to R2U1P8A 
10 meters 
R5 IB3 to Rack 3 
R5IB3P11A to R3U1P9A 
5 meters 
R5 IB3 to Rack 4 
R5IB3P11B to R4U1P10A 
5 meters 
R5 IB3 to Rack 6 
R5IB3P8B to R6U1P4A 
5 meters 
R5 IB3 to Rack 7 
R5IB3P9A to R7U1P5A 
5 meters 
R5 IB3 to Rack 8 
R5IB3P9B to R8U1P6A 
10 meters 
R5 IB2 within Rack 5 
R5IB2P8A to R5U1P3B 
3 meters 
R5 IB2 to Rack 1 
R5IB2P10A to R1U1P7B 
10 meters 
R5 IB2 to Rack 2 
R5IB2P10B to R2U1P8B 
10 meters 
R5 IB2 to Rack 3 
R5IB2P11A to R3U1P9B 
5 meters 
R5 IB2 to Rack 4 
R5IB2P11B to R4U1P10B 
5 meters 
R5 IB2 to Rack 6 
R5IB2P8B to R6U1P4B 
5 meters 
R5 IB2 to Rack 7 
R5IB2P9A to R7U1P5B 
5 meters 
R5 IB2 to Rack 8 
R5IB2P9B to R8U1P6B 
10 meters 
Table 533 shows the cable connections for the sixth spine switch (R6U1) when cabling eight full racks together:
Table 533 Leaf Switch Connections for the Sixth Rack in a EightRack System
Leaf Switch  Connection  Cable Length 

R6 IB3 within Rack 6 
R6IB3P8A to R6U1P3A 
3 meters 
R6 IB3 to Rack 1 
R6IB3P9B to R1U1P6A 
10 meters 
R6 IB3 to Rack 2 
R6IB3P10A to R2U1P7A 
10 meters 
R6 IB3 to Rack 3 
R6IB3P10B to R3U1P8A 
10 meters 
R6 IB3 to Rack 4 
R6IB3P11A to R4U1P9A 
5 meters 
R6 IB3 to Rack 5 
R6IB3P11B to R5U1P10A 
5 meters 
R6 IB3 to Rack 7 
R6IB3P8B to R7U1P4A 
5 meters 
R6 IB3 to Rack 8 
R6IB3P9A to R8U1P5A 
5 meters 
R6 IB2 within Rack 6 
R6IB2P8A to R6U1P3B 
3 meters 
R6 IB2 to Rack 1 
R6IB2P9B to R1U1P6B 
10 meters 
R6 IB2 to Rack 2 
R6IB3P10A to R2U1P7B 
10 meters 
R6 IB2 to Rack 3 
R6IB2P10B to R3U1P8B 
10 meters 
R6 IB2 to Rack 4 
R6IB2P11A to R4U1P9B 
5 meters 
R6 IB2 to Rack 5 
R6IB2P11B to R5U1P10B 
5 meters 
R6 IB2 to Rack 7 
R6IB2P8B to R7U1P4B 
5 meters 
R6 IB2 to Rack 8 
R6IB2P9A to R8U1P5B 
5 meters 
Table 534 shows the cable connections for the seventh spine switch (R7U1) when cabling eight full racks together:
Table 534 Leaf Switch Connections for the Seventh Rack in a EightRack System
Leaf Switch  Connection  Cable Length 

R7 IB3 within Rack 7 
R7IB3P8A to R7U1P3A 
3 meters 
R7 IB3 to Rack 1 
R7IB3P9A to R1U1P5A 
10 meters 
R7 IB3 to Rack 2 
R7IB3P9B to R2U1P6A 
10 meters 
R7 IB3 to Rack 3 
R7IB3P10A to R3U1P7A 
10 meters 
R7 IB3 to Rack 4 
R7IB3P10B to R4U1P8A 
10 meters 
R7 IB3 to Rack 5 
R7IB3P11A to R5U1P9A 
5 meters 
R7 IB3 to Rack 6 
R7IB3P11B to R6U1P10A 
5 meters 
R7 IB3 to Rack 8 
R7IB3P8B to R8U1P4A 
5 meters 
R7 IB2 within Rack 7 
R7IB2P8A to R7U1P3B 
3 meters 
R7 IB2 to Rack 1 
R7IB2P9A to R1U1P5B 
10 meters 
R7 IB2 to Rack 2 
R7IB2P9B to R2U1P6B 
10 meters 
R7 IB2 to Rack 3 
R7IB2P10A to R3U1P7B 
10 meters 
R7 IB2 to Rack 4 
R7IB2P10B to R4U1P8B 
10 meters 
R7 IB2 to Rack 5 
R7IB2P11A to R5U1P9B 
5 meters 
R7 IB2 to Rack 6 
R7IB2P11B to R6U1P10B 
5 meters 
R7 IB2 to Rack 8 
R7IB2P8B to R8U1P4B 
5 meters 
Table 535 shows the cable connections for the eighth spine switch (R8U1) when cabling eight full racks together:
Table 535 Leaf Switch Connections for the Eighth Rack in a EightRack System
Leaf Switch  Connection  Cable Length 

R8 IB3 within Rack 8 
R8IB3P8A to R8U1P3A 
3 meters 
R8 IB3 to Rack 1 
R8IB3P8B to R1U1P4A 
10 meters 
R8 IB3 to Rack 2 
R8IB3P9A to R2U1P5A 
10 meters 
R8 IB3 to Rack 3 
R8IB3P9B to R3U1P6A 
10 meters 
R8 IB3 to Rack 4 
R8IB3P10A to R4U1P7A 
10 meters 
R8 IB3 to Rack 5 
R8IB3P10B to R5U1P8A 
10 meters 
R8 IB3 to Rack 6 
R8IB3P11A to R6U1P9A 
5 meters 
R8 IB3 to Rack 7 
R8IB3P11B to R7U1P10A 
5 meters 
R8 IB2 within Rack 8 
R8IB2P8A to R8U1P3B 
3 meters 
R8 IB2 to Rack 1 
R8IB2P8B to R1U1P4B 
10 meters 
R8 IB2 to Rack 2 
R8IB2P9A to R2U1P5B 
10 meters 
R8 IB2 to Rack 3 
R8IB2P9B to R3U1P6B 
10 meters 
R8 IB2 to Rack 4 
R8IB2P10A to R4U1P7B 
10 meters 
R8 IB2 to Rack 5 
R8IB2P10B to R5U1P8B 
10 meters 
R8 IB2 to Rack 6 
R8IB2P11A to R6U1P9B 
5 meters 
R8 IB2 to Rack 7 
R8IB2P11B to R7U1P10B 
5 meters 