Overview of Adding Another Rack to an Existing System

Review the following notes before cabling racks together.

• The procedures for extending racks with RoCE Network Fabric (X8M and later) are different than the procedures for racks with InfiniBand Network Fabric (X8 and earlier.)

• Racks with InfiniBand Network Fabric can be cabled together with no downtime. Depending on the procedure being used, racks with RoCE Network Fabric might require downtime when cabling racks together.

• Cabling within a live network must be done carefully in order to avoid potentially serious disruptions.

• There can be performance degradation while cabling the racks together. This degradation results from data retransmission due to packet loss and reduced network bandwidth when a cable is unplugged.

• Redundancy with the RDMA Network Fabric can be compromised while cabling the racks together. This occurs whenever the RDMA Network Fabric ports or switches are taken offline and all traffic must use the remaining switches.

• Only the existing racks are operational when adding racks. It is assumed that the servers on any new racks are initially powered down.

• The software running on the systems cannot have problems related to RDMA Network Fabric restarts. To verify the configuration, run infinicheck separately on each rack before connecting multiple racks together.

• It is assumed that each ZDLRA Rack has three RDMA Network Fabric switches already installed.

• The new racks have been configured with the appropriate IP addresses to be migrated into the expanded system prior to any cabling, and there are no duplicate IP addresses.

• Racks with RoCE Network Fabric use one loopback IP interface on each spine switch and two loopback IP interfaces on each leaf switch. The IP addressing scheme uses IANA ‘Shared Address Space’ 100.64.0.0/10. This ensures that there is no overlap with IPv4 addresses in the network using other schemes.

  • Leaf loopback0 IPs are assigned as 100.64.0.101, 100.64.0.102, 100.64.0.103, and so on.

  • Leaf loopback1 IPs are assigned as 100.64.1.101, 100.64.1.102, 100.64.1.103, and so on.

  • Spine loopback0 IPs are assigned as 100.64.0.201, 100.64.0.202, up to 100.64.0.208.