Root Domains

A Root Domain is an SR-IOV domain that hosts the physical I/O devices, or physical functions (PFs), such as the IB HCAs and 10GbE NICs installed in the PCIe slots. Almost all of its CPU and memory resources are parked for later use by I/O Domains. Logical devices, or virtual functions (VFs), are created from each PF, with each PF hosting 32 VFs.

Because Root Domains host the physical I/O devices, just as dedicated domains currently do, Root Domains essentially exist at the same level as dedicated domains.

With the introduction of Root Domains, the following parts of the domain configuration for a SuperCluster are set at the time of the initial installation and can only be changed by an Oracle representative:

• Type of domain:

  • Root Domain

  • Application Domain running Oracle Solaris 10 (dedicated domain)

  • Application Domain running Oracle Solaris 11 (dedicated domain)

  • Database Domain (dedicated domain)

• Number of Root Domains and dedicated domains on the server

A domain can only be a Root Domain if it has either one or two IB HCAs associated with it. A domain cannot be a Root Domain if it has more than two IB HCAs associated with it. If you have a domain that has more than two IB HCAs associated with it (for example, the H1-1 domain in an Oracle SuperCluster T5-8), then that domain must be a dedicated domain.

When deciding which domains will be a Root Domain, the last domain must always be the first Root Domain, and you would start from the last domain in your configuration and go in for every additional Root Domain. For example, assume you have four domains in your configuration, and you want two Root Domains and two dedicated domains. In this case, the first two domains would be dedicated domains and the last two domains would be Root Domains.

The following domains have only one or two IB HCAs associated with them and can therefore be used as a Root Domain:

• Small Domains (one IB HCA)

• Medium Domains (two IB HCAs)

In addition, the first domain in the system (the Control Domain) will always be a dedicated domain. The Control Domain cannot be a Root Domain. Therefore, you cannot have all of the domains on your server as Root Domains, but you can have a mixture of Root Domains and dedicated domains on your server or all of the domains as dedicated domains.

A certain amount of CPU core and memory is always reserved for each Root Domain, depending on which domain is being used as a Root Domain in the domain configuration and the number of IB HCAs and 10GbE NICs that are associated with that Root Domain:

• The last domain in a domain configuration:

  • Two cores and 32 GB of memory reserved for a Root Domain with one IB HCA and 10GbE NIC

  • Four cores and 64 GB of memory reserved for a Root Domain with two IB HCAs and 10GbE NICs

• Any other domain in a domain configuration:

  • One core and 16 GB of memory reserved for a Root Domain with one IB HCA and 10GbE NIC

  • Two cores and 32 GB of memory reserved for a Root Domain with two IB HCAs and 10GbE NICs

The remaining CPU core and memory resources associated with each Root Domain are parked in CPU and memory repositories, as shown in the following graphic.

CPU and memory repositories contain resources not only from the Root Domains, but also any parked resources from the dedicated domains. Whether CPU core and memory resources originated from dedicated domains or from Root Domains, once those resources have been parked in the CPU and memory repositories, those resources are no longer associated with their originating domain. These resources become equally available to I/O Domains.

In addition, CPU and memory repositories contain parked resources only from the compute server that contains the domains providing those parked resources. In other words, if you have two compute servers and both compute servers have Root Domains, there would be two sets of CPU and memory repositories, where each compute server would have its own CPU and memory repositories with parked resources.

For example, assume you have four domains on your compute server, with three of the four domains as Root Domains, as shown in the previous graphic. Assume each domain has the following IB HCAs and 10GbE NICs, and the following CPU core and memory resources:

• One IB HCA and one 10GbE NIC

• 16 cores

• 256 GB of memory

In this situation, the following CPU core and memory resources are reserved for each Root Domain, with the remaining resources available for the CPU and memory repositories:

• Two cores and 32 GB of memory reserved for the last Root Domains in this configuration. 14 cores and 224 GB of memory available from this Root Domain for the CPU and memory repositories.

• One core and 16 GB of memory reserved for the second and third Root Domains in this configuration.

  • 15 cores and 240 GB of memory available from each of these Root Domains for the CPU and memory repositories.

  • A total of 30 cores (15 x 2) and 480 GB of memory (240 GB x 2) available for the CPU and memory repositories from these two Root Domains.

A total of 44 cores (14 + 30 cores) are therefore parked in the CPU repository, and 704 GB of memory (224 + 480 GB of memory) are parked in the memory repository and are available for the I/O Domains.

With Root Domains, connections to the 10GbE client access network go through the physical ports on each 10GbE NIC, and connections to the IB network go through the physical ports on each IB HCA, just as they did with dedicated domains. However, cards used with Root Domains must also be SR-IOV compliant. SR-IOV compliant cards enable VFs to be created on each card, where the virtualization occurs in the card itself.

The VFs from each Root Domain are parked in the IB VF and 10GbE VF repositories, similar to the CPU and memory repositories, as shown in the following graphic.

Even though the VFs from each Root Domain are parked in the VF repositories, the VFs are created on each 10GbE NIC and IB HCA, so those VFs are associated with the Root Domain that contains those specific 10GbE NIC and IB HCA cards. For example, looking at the example configuration in the previous graphic, the VFs created on the last (right most) 10GbE NIC and IB HCA will be associated with the last Root Domain.