Multi-Rack Cabling Overview

You connect multiple Oracle SuperCluster T5-8 systems together using three Sun Datacenter InfiniBand Switch 36 switches. Oracle SuperCluster T5-8 includes three Sun Datacenter InfiniBand Switch 36 switches. These switches attach to standard Quad Small Form-factor Pluggable (QSFP) connectors at the end of the InfiniBand cables. The procedures in this section assume the racks are adjacent to each other. If they are not, then longer cables may be required for the connections.

The switch at rack unit 1 (U1) is referred to as the spine switch. The switches at rack unit 26 (U26) and rack unit 32 (U32) are referred to as leaf switches. In a single rack, the two leaf switches are interconnected using seven connections. In addition, each leaf switch has one connection to the spine switch. The leaf switches connect to the spine switch as shown in the following graphic.

When connecting multiple racks together, remove the seven existing inter-switch connections between each leaf switches, as well as the two connections between the leaf switches and the spine switch. From each leaf switch, distribute eight connections over the spine switches in all racks. In multi-rack environments, the leaf switches inside a rack are no longer directly interconnected, as shown in the following graphic.

As shown in the preceding graphic, each leaf switch in rack 1 connects to the following switches:

• Four connections to its internal spine switch

• Four connections to the spine switch in rack 2

The spine switch in rack 1 connects to the following switches:

• Eight connections to both internal leaf switches

• Eight connections to both leaf switches in rack 2

In Oracle SuperCluster T5-8, the spine and leaf switches are installed in the following locations:

• Spine switch in U1

• Two leaf switches in U26 and U32