Clustering Considerations for InfiniBand
Like a network built on top of Ethernet devices, an InfiniBand network needs to be part of a redundant fabric topology in order to guard against network failures inside and outside of the appliance. The network topology should include IPMP to protect against network failures at the link level with a broader plan for redundancy for HCAs, switches and subnet managers.
Figure 5 Clustering Considerations for InfiniBand
To ensure proper cluster configuration, each controller must be populated with identical HCAs in identical slots. Furthermore, each corresponding HCA port must be configured into the same partition (pkey) on the subnet manager with identical membership privileges and attached to the same network. To reduce complexity and ensure proper redundancy, it is recommended that each port belong to only one partition in the InfiniBand sub-network. Network interfaces may be configured as either singleton or private resources, provided they have static IP configuration. When configured as a singleton resource, all of the IB partition datalinks and devices used to construct an interface may be active on only one controller at any given time. A concrete example of this is shown in the illustration above. Changes to partition membership for corresponding ports must happen at the same time and in a manner consistent with the clustering rules described earlier. Your qualified Oracle reseller or service representative can assist in planning hardware upgrades that will meet these requirements.
The following illustration shows cluster configuration for subnet manager redundancy. Greater redundancy is achieved by connecting two dual-port HCAs to a redundant pair of server switches.
Figure 6 Cluster Configuration for Subnet Manager Redundancy
Related Topics