InfiniBand Network Overview

Language:

The InfiniBand network connects the SPARC T5-8 servers, ZFS storage appliance, and Exadata Storage Servers using the InfiniBand switches on the rack. This non-routable network is fully contained in Oracle SuperCluster T5-8, and does not connect to your existing network.

When Oracle SuperCluster T5-8 is configured with the appropriate types of domains, the InfiniBand network is partitioned to define the data paths between the SPARC T5-8 servers, and between the SPARC T5-8 servers and the storage appliances.

The defined InfiniBand data path coming out of the SPARC T5-8 servers varies, depending on the type of domain created on each SPARC T5-8 server:

InfiniBand Network Data Paths for a Database Domain

Note - The information in this section applies to a Database Domain that is either a dedicated domain or a Database I/O Domain.

When a Database Domain is created on a SPARC T5-8 server, the Database Domain has the following InfiniBand paths:

SPARC T5-8 server to both Sun Datacenter InfiniBand Switch 36 leaf switches
SPARC T5-8 server to each Exadata Storage Server, through the Sun Datacenter InfiniBand Switch 36 leaf switches
SPARC T5-8 server to the ZFS storage appliance, through the Sun Datacenter InfiniBand Switch 36 leaf switches

The number of InfiniBand HCAs that are assigned to the Database Domain varies, depending on the type of configuration that is set up on the SPARC T5-8 server.

For the InfiniBand HCAs assigned to a Database Domain, the following InfiniBand private networks are used:

Storage private network: One InfiniBand private network for the Database Domains to communicate with each other and with the Application Domains running Oracle Solaris 10, and with the ZFS storage appliance
Exadata private network: One InfiniBand private network for the Oracle Database 11g Real Application Clusters (Oracle RAC) interconnects, and for communication between the Database Domains and the Exadata Storage Servers

The two ports on each InfiniBand HCA connect to different Sun Datacenter InfiniBand Switch 36 leaf switches to provide redundancy between the SPARC T5-8 servers and the leaf switches. For more information on the physical connections for the SPARC T5-8 servers to the leaf switches, see InfiniBand Private Network Physical Connections (SPARC T5-8 Servers).

InfiniBand Network Data Paths for an Application Domain

Note - The information in this section applies to an Application Domain that is either a dedicated domain or an Application I/O Domain.

When an Application Domain is created on a SPARC T5-8 server (an Application Domain running either Oracle Solaris 10 or Oracle Solaris 11), the Application Domain has the following InfiniBand paths:

SPARC T5-8 server to both Sun Datacenter InfiniBand Switch 36 leaf switches
SPARC T5-8 server to the ZFS storage appliance, through the Sun Datacenter InfiniBand Switch 36 leaf switches

Note that the Application Domain would not access the Exadata Storage Servers, which are used only for the Database Domain.

The number of InfiniBand HCAs that are assigned to the Application Domain varies, depending on the type of configuration that is set up on the SPARC T5-8 server.

For the InfiniBand HCAs assigned to an Application Domain, the following InfiniBand private networks are used:

Storage private network: One InfiniBand private network for Application Domains to communicate with each other and with the Database Domains, and with the ZFS storage appliance
Oracle Solaris Cluster private network: Two InfiniBand private networks for the optional Oracle Solaris Cluster interconnects

The two ports on each InfiniBand HCA will connect to different Sun Datacenter InfiniBand Switch 36 leaf switches to provide redundancy between the SPARC T5-8 servers and the leaf switches. For more information on the physical connections for the SPARC T5-8 servers to the leaf switches, see InfiniBand Private Network Physical Connections (SPARC T5-8 Servers).