Multi-Rack Cabling Overview

Three Sun Datacenter InfiniBand Switch 36 switches are needed when cabling racks together. The SPARC SuperCluster T4-4 includes three Sun Datacenter InfiniBand Switch 36 switches. These switches attach to standard Quad Small Form-factor Pluggable (QSFP) connectors at the end of the InfiniBand cables. The procedures in this section assume the racks are adjacent to each other. If they are not, then longer cables may be required for the connections.

The switch at rack unit 1 (U1) is referred to as the spine switch. The switches at rack unit 18 (U18) and rack unit 24 (U24) are referred to as leaf switches. In a single rack, the two leaf switches are interconnected using seven connections. In addition, each leaf switch has one connection to the spine switch. The leaf switches connect to the spine switch as shown in the following graphic.

When connecting multiple racks together, remove the seven existing inter-switch connections between each leaf switches, as well as the two connections between the leaf switches and the spine switch. From each leaf switch, distribute eight connections over the spine switches in all racks. In multi-rack environments, the leaf switches inside a rack are no longer directly interconnected, as shown in the following graphic.

As shown in the preceding graphic, each leaf switch in rack 1 connects to the following switches:

• Four connections to its internal spine switch

• Four connections to the spine switch in rack 2

The spine switch in rack 1 connects to the following switches:

• Eight connections to both internal leaf switches

• Eight connections to both leaf switches in rack 2

In the SPARC SuperCluster T4-4, the spine and leaf switches are installed in the following locations:

• Spine switch in U1

• Two leaf switches in U18 and U24