The InfiniBand network connects the SPARC T4-4 servers, Sun ZFS Storage 7320 appliance, and Exadata Storage Servers using the InfiniBand switches on the rack. This non-routable network is fully contained in SPARC SuperCluster T4-4, and does not connect to your existing network.
When the SPARC SuperCluster T4-4 is configured with the appropriate types of domains, the InfiniBand network is partitioned to define the data paths between the SPARC T4-4 servers, and between the SPARC T4-4 servers and the storage appliances.
The defined InfiniBand data path coming out of the SPARC T4-4 servers varies, depending on the type of domain created on each SPARC T4-4 server:
When a Database Domain is created on a SPARC T4-4 server, the Database Domain has the following InfiniBand paths:
SPARC T4-4 server to both Sun Datacenter InfiniBand Switch 36 leaf switches
SPARC T4-4 server to each Exadata Storage Server, through the Sun Datacenter InfiniBand Switch 36 leaf switches
SPARC T4-4 server to the Sun ZFS Storage 7320 appliance, through the Sun Datacenter InfiniBand Switch 36 leaf switches
Each SPARC T4-4 server contains four InfiniBand HCAs. The number of InfiniBand HCAs that are assigned to the Database Domain varies, depending on the type of configuration that is set up on the SPARC T4-4 server.
For the InfiniBand HCAs assigned to a Database Domain, the following InfiniBand private networks are used:
One InfiniBand private network for the Database Domains to communicate with each other and with the Application Domains running Oracle Solaris 10, and with the Sun ZFS Storage 7320 appliance
One InfiniBand private network for the Oracle Database 11g Real Application Clusters (Oracle RAC) interconnects, and for communication between the Database Domains and the Exadata Storage Servers
The two ports on each InfiniBand HCA will connect to different Sun Datacenter InfiniBand Switch 36 leaf switches to provide redundancy between the SPARC T4-4 servers and the leaf switches. For more information on the physical connections for the SPARC T4-4 servers to the leaf switches, see InfiniBand Private Network Physical Connections (SPARC T4-4 Servers).
When an Application Domain is created on a SPARC T4-4 server (an Application Domain running either Oracle Solaris 10 or Oracle Solaris 11), the Application Domain will have the following InfiniBand paths:
SPARC T4-4 server to both Sun Datacenter InfiniBand Switch 36 leaf switches
SPARC T4-4 server to the Sun ZFS Storage 7320 appliance, through the Sun Datacenter InfiniBand Switch 36 leaf switches
Note that the Application Domain would not access the Exadata Storage Servers, which are used only for the Database Domain.
Every SPARC T4-4 server contains four InfiniBand HCAs. The number of InfiniBand HCAs that are assigned to the Application Domain varies, depending on the type of configuration that is set up on the SPARC T4-4 server.
For the InfiniBand HCAs assigned to an Application Domain, the following InfiniBand private networks are used:
One InfiniBand private network for Application Domains to communicate with each other and with the Database Domains, and with the Sun ZFS Storage 7320 appliance
Two InfiniBand private networks for the optional Oracle Solaris Cluster interconnects
The two ports on each InfiniBand HCA will connect to different Sun Datacenter InfiniBand Switch 36 leaf switches to provide redundancy between the SPARC T4-4 servers and the leaf switches. For more information on the physical connections for the SPARC T4-4 servers to the leaf switches, see InfiniBand Private Network Physical Connections (SPARC T4-4 Servers).