Clustering Considerations for InfiniBand
Note:
Oracle ZFS Storage ZS9-2 controllers do not support InfiniBand.Like a network built on top of Ethernet devices, an InfiniBand network needs to be part of a redundant fabric topology in order to guard against network failures inside and outside of the appliance. The network topology should include IPMP to protect against network failures at the link level with a broader plan for redundancy for HCAs, switches, and subnet managers.
To ensure proper cluster configuration, each controller must be populated with identical HCAs in identical slots. Furthermore, each corresponding HCA port must be configured into the same partition (pkey) on the subnet manager with identical membership privileges and attached to the same network. To reduce complexity and ensure proper redundancy, it is recommended that each port belong to only one partition in the InfiniBand sub-network. Network interfaces may be configured as either singleton or private resources, provided they have static IP configuration. When configured as a singleton resource, all of the InfiniBand partition datalinks and devices used to construct an interface may be active on only one controller at any given time. An example of this is shown in the following figure.
Changes to partition membership for corresponding ports must happen at the same time. Your qualified Oracle reseller or service representative can assist in planning hardware upgrades that will meet these requirements.
The following figure shows cluster configuration for subnet manager redundancy. Greater redundancy is achieved by connecting two dual-port HCAs to a redundant pair of server switches.
Related Topics