Six-Rack Cabling with InfiniBand Network Fabric
Note the following when cabling six racks together with InfiniBand Network Fabric:
-
The spine switch is in U1 (also referred to as IB1) for all racks.
-
In the following tables, the leaf switches are referred to as IB2 and IB3. Use the rack unit that is appropriate for your system.
-
The cable lengths shown in the tables assume the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cables lengths. Up to 100 meters is supported.
-
Only optical cables are supported for lengths greater than 5 meters.
Table 24-15 shows the cable connections for the first spine switch (R1-U1) when cabling six racks together:
Table 24-15 Leaf Switch Connections for the First Rack in a Six-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
| R1 IB3 within Rack 1 | R1-IB3-P8A to R1-U1-P3A R1-IB3-P8B to R1-U1-P4A |
3 meters |
| R1 IB3 to Rack 2 | R1-IB3-P9A to R2-U1-P5A R1-IB3-P9B to R2-U1-P6A |
5 meters |
| R1 IB3 to Rack 3 | R1-IB3-P10A to R3-U1-P7A | 5 meters |
| R1 IB3 to Rack 4 | R1-IB3-P10B to R4-U1-P8A | 10 meters |
| R1 IB3 to Rack 5 | R1-IB3-P11A to R5-U1-P9A | 10 meters |
| R1 IB3 to Rack 6 | R1-IB3-P11B to R6-U1-P10A | 10 meters |
| R1 IB2 within Rack 1 | R1-IB2-P8A to R1-U1-P3B R1-IB2-P8B to R1-U1-P4B |
3 meters |
| R1 IB2 to Rack 2 | R1-IB2-P9A to R2-U1-P5B R1-IB2-P9B to R2-U1-P6B |
5 meters |
| R1 IB2 to Rack 3 | R1-IB2-P10A to R3-U1-P7B | 5 meters |
| R1 IB2 to Rack 4 | R1-IB2-P10B to R4-U1-P8B | 10 meters |
| R1 IB2 to Rack 5 | R1-IB2-P11A to R5-U1-P9B | 10 meters |
| R1 IB2 to Rack 6 | R1-IB2-P11B to R6-U1-P10B | 10 meters |
Table 24-16 shows the cable connections for the second spine switch (R2-U1) when cabling six full racks together:
Table 24-16 Leaf Switch Connections for the Second Rack in a Six-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
| R2 IB3 within Rack 2 | R2-IB3-P8A to R2-U1-P3A R2-IB3-P8B to R2-U1-P4A |
3 meters |
| R2 IB3 to Rack 1 | R2-IB3-P11B to R1-U1-P10A | 5 meters |
| R2 IB3 to Rack 3 | R2-IB3-P9A to R3-U1-P5A R2-IB3-P9B to R3-U1-P6A |
5 meters |
| R2 IB3 to Rack 4 | R2-IB3-P10A to R4-U1-P7A | 5 meters |
| R2 IB3 to Rack 5 | R2-IB3-P10B to R5-U1-P8A | 10 meters |
| R2 IB3 to Rack 6 | R2-IB3-P11Ato R6-U1-P9A | 10 meters |
| R2 IB2 within Rack 2 | R2-IB2-P8A to R2-U1-P3B R2-IB2-P8B to R2-U1-P4B |
3 meters |
| R2 IB2 to Rack 1 | R2-IB2-P11B to R1-U1-P10B | 5 meters |
| R2 IB2 to Rack 3 | R2-IB2-P9A to R3-U1-P5B R2-IB2-P9B to R3-U1-P6B |
5 meters |
| R2 IB2 to Rack 4 | R2-IB2-P10A to R4-U1-P7B | 5 meters |
| R2 IB2 to Rack 5 | R2-IB2-P10B to R5-U1-P8B | 10 meters |
| R2 IB2 to Rack 6 | R2-IB2-P11Ato R6-U1-P9B | 10 meters |
Table 24-17 shows the cable connections for the third spine switch (R3-U1) when cabling six full racks together:
Table 24-17 Leaf Switch Connections for the Third Rack in a Six-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
| R3 IB3 within Rack 3 | R3-IB3-P8A to R3-U1-P3A R3-IB3-P8B to R3-U1-P4A |
3 meters |
| R3 IB3 to Rack 1 | R3-IB3-P11A to R1-U1-P9A | 5 meters |
| R3 IB3 to Rack 2 | R3-IB3-P11B to R2-U1-P10A | 5 meters |
| R3 IB3 to Rack 4 | R3-IB3-P9A to R4-U1-P5A R3-IB3-P9B to R4-U1-P6A |
5 meters |
| R3 IB3 to Rack 5 | R3-IB3-P10A to R5-U1-P7A | 5 meters |
| R3 IB3 to Rack 6 | R3-IB3-P10B to R6-U1-P8A | 10 meters |
| R3 IB2 within Rack 3 | R3-IB2-P8A to R3-U1-P3B R3-IB2-P8B to R3-U1-P4B |
3 meters |
| R3 IB2 to Rack 1 | R3-IB2-P11A to R1-U1-P9B | 5 meters |
| R3 IB2 to Rack 2 | R3-IB2-P11B to R2-U1-P10B | 5 meters |
| R3 IB2 to Rack 4 | R3-IB2-P9A to R4-U1-P5B R3-IB2-P9B to R4-U1-P6B |
5 meters |
| R3 IB2 to Rack 5 | R3-IB2-P10A to R5-U1-P7B | 5 meters |
| R3 IB2 to Rack 6 | R3-IB2-P10B to R6-U1-P8B | 10 meters |
Table 24-18 shows the cable connections for the fourth spine switch (R4-U1) when cabling six full racks together:
Table 24-18 Leaf Switch Connections for the Fourth Rack in a Six-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
| R4 IB3 within Rack 4 | R4-IB3-P8A to R4-U1-P3A R4-IB3-P8B to R4-U1-P4A |
3 meters |
| R4 IB3 to Rack 1 | R4-IB3-P10B to R1-U1-P8A | 10 meters |
| R4 IB3 to Rack 2 | R4-IB3-P11A to R2-U1-P9A | 5 meters |
| R4 IB3 to Rack 3 | R4-IB3-P11B to R3-U1-P10A | 5 meters |
| R4 IB3 to Rack 5 | R4-IB3-P9A to R5-U1-P5A R4-IB3-P9B to R5-U1-P6A |
5 meters |
| R4 IB3 to Rack 6 | R4-IB3-P10A to R6-U1-P7A | 5 meters |
| R4 IB2 within Rack 4 | R4-IB2-P8A to R4-U1-P3B R4-IB2-P8B to R4-U1-P4B |
3 meters |
| R4 IB2 to Rack 1 | R4-IB2-P10B to R1-U1-P8B | 10 meters |
| R4 IB2 to Rack 2 | R4-IB2-P11A to R2-U1-P9B | 5 meters |
| R4 IB2 to Rack 3 | R4-IB2-P11B to R3-U1-P10B | 5 meters |
| R4 IB2 to Rack 5 | R4-IB2-P9A to R5-U1-P5B R4-IB2-P9B to R5-U1-P6B |
5 meters |
| R4 IB2 to Rack 6 | R4-IB2-P10A to R6-U1-P7B | 5 meters |
Table 24-19 shows the cable connections for the fifth spine switch (R5-U1) when cabling six full racks together:
Table 24-19 Leaf Switch Connections for the Fifth Rack in a Six-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
| R5 IB3 within Rack 5 | R5-IB3-P8A to R5-U1-P3A R5-IB3-P8B to R5-U1-P4A |
3 meters |
| R5 IB3 to Rack 1 | R5-IB3-P10A to R1-U1-P7A | 10 meters |
| R5 IB3 to Rack 2 | R5-IB3-P10B to R2-U1-P8A | 10 meters |
| R5 IB3 to Rack 3 | R5-IB3-P11A to R3-U1-P9A | 5 meters |
| R5 IB3 to Rack 4 | R5-IB3-P11B to R4-U1-P10A | 5 meters |
| R5 IB3 to Rack 6 | R5-IB3-P9A to R6-U1-P5A R5-IB3-P9B to R6-U1-P6A |
5 meters |
| R5 IB2 within Rack 5 | R5-IB2-P8A to R5-U1-P3B R5-IB2-P8B to R5-U1-P4B |
3 meters |
| R5 IB2 to Rack 1 | R5-IB2-P10A to R1-U1-P7B | 10 meters |
| R5 IB2 to Rack 2 | R5-IB2-P10B to R2-U1-P8B | 10 meters |
| R5 IB2 to Rack 3 | R5-IB2-P11A to R3-U1-P9B | 5 meters |
| R5 IB2 to Rack 4 | R5-IB2-P11B to R4-U1-P10B | 5 meters |
| R5 IB2 to Rack 6 | R5-IB2-P9A to R6-U1-P5B R5-IB2-P9B to R6-U1-P6B |
5 meters |
Table 24-20 shows the cable connections for the sixth spine switch (R6-U1) when cabling six full racks together:
Table 24-20 Leaf Switch Connections for the Sixth Rack in a Six-Rack System
| Leaf Switch | Connection | Cable Length |
|---|---|---|
| R6 IB3 within Rack 6 | R6-IB3-P8A to R6-U1-P3A R6-IB3-P8B to R6-U1-P4A |
3 meters |
| R6 IB3 to Rack 1 | R6-IB3-P9A to R1-U1-P5A R6-IB3-P9B to R1-U1-P6A |
10 meters |
| R6 IB3 to Rack 2 | R6-IB3-P10A to R2-U1-P7A | 10 meters |
| R6 IB3 to Rack 3 | R6-IB3-P10B to R3-U1-P8A | 10 meters |
| R6 IB3 to Rack 4 | R6-IB3-P11A to R4-U1-P9A | 5 meters |
| R6 IB3 to Rack 5 | R6-IB3-P11B to R5-U1-P10A | 5 meters |
| R6 IB2 within Rack 6 | R6-IB2-P8A to R6-U1-P3B R6-IB2-P8B to R6-U1-P4B |
3 meters |
| R6 IB2 to Rack 1 | R6-IB2-P9A to R1-U1-P5B R6-IB2-P9B to R1-U1-P6B |
10 meters |
| R6 IB2 to Rack 2 | R6-IB2-P10A to R2-U1-P7B | 10 meters |
| R6 IB2 to Rack 3 | R6-IB2-P10B to R3-U1-P8B | 10 meters |
| R6 IB2 to Rack 4 | R6-IB2-P11A to R4-U1-P9B | 5 meters |
| R6 IB2 to Rack 5 | R6-IB2-P11B to R5-U1-P10B | 5 meters |