Switch-Mode Configuration Considerations

Before connecting the datacenters, be aware of the following:

• Switch mode uses the generic minhop routing algorithm to provide efficient, deadlock-free routing. To guarantee no deadlock conditions occur, the topology must ensure that the shortest paths between any communicating end nodes never imply that two or more flows all travel in the same direction in a ring of four or more connected switches.

• To prevent deadlock, ensure that different flows (for example, data traffic) that can contribute to deadlocks (for example, in combination with other data traffic) will use different Virtual Lanes (VLs).

• If you are using IPoIB in your deployment, you must define the VL independence by configuring specific IP subnets that map to IB service levels (SLs). See Virtual Lane Independence Through IP Subnetting.

  Any host-to-host data traffic that is not based on IPoIB must use the same SLs as documented in Virtual Lane Independence Through IP Subnetting.

• If the virtualization switches are used for standard server I/O as well as IB LR connectivity, the I/O traffic is limited to the local datacenter and cannot cross any IB LR link to the other datacenter. The I/O traffic must use the same SL or VL as is used for host-to-host communication within the local site.

• Oracle Fabric Manager can be used for OOB management. Oracle Fabric Manager requires specific ports be available to manage both sides of the LR links (for example, in case a firewall exists on the management network between the two datacenters). This table shows the ports OFM requires.

  Port Description 8443 (HTTPS) Used for configured users or admin user access to Oracle Fabric Manager. As an alternative, you can leave those ports closed, but open an RDP session and use the local browser for added security. 443 Used to configure Oracle Fabric Interconnects and Oracle SDN Controllers. 7443 Used to configure Oracle IB switches. 7777 Used to discover Oracle IB switches. 22 Used to download logs, and export and import backup files on all types of devices.

• In addition to the OFM requirements, the OOB coordination between subnet managers on each side of the IB LR link requires:

  • Port 111 is available and not assigned to any other traffic. Port 111 is used for establishing RPC connections.

  • Dynamically allocating port numbers for RPC connections must be allowed also.

• If you will use IB performance monitoring tools in the switch-mode topology, be aware that "global' gathering of performance management data is not supported. Instead, use these options:

  • Performance data and other IB-counter information should be retrieved OOB through the leaf or virtualization switch's management interfaces.

  • Standard IB tools can be used to collect counter information from one node at a time, but do not collect data from all nodes in a single operation. Only one statistics-gathering operation can be active at a time.