6.3 Three-Rack Cabling with InfiniBand Network Fabric
Note the following when cabling three racks together with InfiniBand Network Fabric:
-
The spine switch is in U1 (also referred to as IB1) for all racks.
-
In the following tables, the leaf switches are referred to as IB2 and IB3. Use the rack unit that is appropriate for your system.
-
In Oracle Exadata Database Machine X2-2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers, the leaf switches are in U20 and U24, also referred to as IB2 and IB3, respectively.
-
In Oracle Exadata Database Machine X2-8 and later, the leaf switches are in U21 and U23, also referred to as IB2 and IB3, respectively.
-
In Oracle Exadata Database Machine X4-2 and later racks, or Oracle Exadata Database Machine X3-2 Full Racks or Oracle Exadata Storage Expansion Rack X3-2 racks, the leaf switches are in U20 and U22, also referred to as IB2 and IB3, respectively.
-
The cable lengths shown in the tables assume the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cables lengths. Up to 100 meters is supported.
-
Only optical cables are supported for lengths greater than 5 meters.
Table 6-3 shows the cable connections for the first spine switch (R1-U1) when cabling three racks together:
Table 6-3 Leaf Switch Connections for the First Rack in a Three-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R1 IB3 within Rack 1 |
R1-IB3-P8A to R1-U1-P3A R1-IB3-P8B to R1-U1-P4A R1-IB3-P9A to R1-U1-P5A |
3 meters |
|
R1 IB3 to Rack 2 |
R1-IB3-P9B to R2-U1-P6A R1-IB3-P10A to R2-U1-P7A R1-IB3-P10B to R2-U1-P8A |
5 meters |
|
R1 IB3 to Rack 3 |
R1-IB3-P11A to R3-U1-P9A R1-IB3-P11B to R3-U1-P10A |
5 meters |
|
R1 IB2 within Rack 1 |
R1-IB2-P8A to R1-U1-P3B R1-IB2-P8B to R1-U1-P4B R1-IB2-P9A to R1-U1-P5B |
3 meters |
|
R1 IB2 to Rack 2 |
R1-IB2-P9B to R2-U1-P6B R1-IB2-P10A to R2-U1-P7B R1-IB2-P10B to R2-U1-P8B |
5 meters |
|
R1 IB2 to Rack 3 |
R1-IB2-P11A to R3-U1-P9B R1-IB2-P11B to R3-U1-P10B |
5 meters |
Table 6-4 shows the cable connections for the second spine switch (R2-U1) when cabling three racks together:
Table 6-4 Leaf Switch Connections for the Second Rack in a Three-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R2 IB3 within Rack 2 |
R2-IB3-P8A to R2-U1-P3A R2-IB3-P8B to R2-U1-P4A R2-IB3-P9A to R2-U1-P5A |
3 meters |
|
R2 IB3 to Rack 1 |
R2-IB3-P11A to R1-U1-P9A R2-IB3-P11B to R1-U1-P10A |
5 meters |
|
R2 IB3 to Rack 3 |
R2-IB3-P9B to R3-U1-P6A R2-IB3-P10A to R3-U1-P7A R2-IB3-P10B to R3-U1-P8A |
5 meters |
|
R2 IB2 within Rack 2 |
R2-IB2-P8A to R2-U1-P3B R2-IB2-P8B to R2-U1-P4B R2-IB2-P9A to R2-U1-P5B |
3 meters |
|
R2 IB2 to Rack 1 |
R2-IB2-P11A to R1-U1-P9B R2-IB2-P11B to R1-U1-P10B |
5 meters |
|
R2 IB2 to Rack 3 |
R2-IB2-P9B to R3-U1-P6B R2-IB2-P10A to R3-U1-P7B R2-IB2-P10B to R3-U1-P8B |
5 meters |
Table 6-5 shows the cable connections for the third spine switch (R3-U1) when cabling three full racks together:
Table 6-5 Leaf Switch Connections for the Third Rack in a Three-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
|
R3 IB3 within Rack 3 |
R3-IB3-P8A to R3-U1-P3A R3-IB3-P8B to R3-U1-P4A R3-IB3-P9A to R3-U1-P5A |
3 meters |
|
R3 IB3 to Rack 1 |
R3-IB3-P9B to R1-U1-P6A R3-IB3-P10A to R1-U1-P7A R3-IB3-P10B to R1-U1-P8A |
5 meters |
|
R3 IB3 to Rack 2 |
R3-IB3-P11A to R2-U1-P9A R3-IB3-P11B to R2-U1-P10A |
5 meters |
|
R3 IB2 within Rack 3 |
R3-IB2-P8A to R3-U1-P3B R3-IB2-P8B to R3-U1-P4B R3-IB2-P9A to R3-U1-P5B |
3 meters |
|
R3 IB2 to Rack 1 |
R3-IB2-P9B to R1-U1-P6B R3-IB2-P10A to R1-U1-P7B R3-IB2-P10B to R1-U1-P8B |
5 meters |
|
R3 IB2 to Rack 2 |
R3-IB2-P11A to R2-U1-P9B R3-IB2-P11B to R2-U1-P10B |
5 meters |