Root Domains
The sole purpose of Root Domains is to provide shared I/O resources to user-created domains (I/O Domains). Like dedicated domains, root domains are created during the installation by Oracle personnel. Root Domains own InfiniBand HCA and 10 GbE NIC resources (one of each per root domain), and in some cases an optional fibre channel HBA. Unlike dedicated domains, Root Domains do not host user applications. For that reason they are assigned only a small allocation of CPU and memory. Additional HCAs and NICs cannot be added to Root Domains.
When deciding which domains will be a Root Domain, the last domain must always be the first Root Domain, and the remaining domains can be any combination of Root Domains or dedicated domains. However, a domain can only be a Root Domain if it has either one or two IB HCAs associated with it. A domain cannot be a Root Domain if it has more than two IB HCAs associated with it. If a domain has more than two IB HCAs associated with it.
Note - For SuperCluster M8, only one HCA is supported in a Root Domain.
Note - Even though a domain with two IB HCAs is valid for a Root Domain, domains with only one IB HCA should be used as Root Domains. When a Root Domain has a single IB HCA, fewer I/O Domains have dependencies on the I/O devices provided by that Root Domain. Flexibility around high availability also increases with Root Domains with one IB HCA.
A certain amount of CPU and memory resources are reserved for each Root Domain, depending on which domain is being used as a Root Domain in the domain configuration and the number of IB HCAs and 10GbE NICs that are associated with that Root Domain:
-
The last domain in a domain configuration:
-
Two cores and 32 GB of memory reserved for a Root Domain with one IB HCA and 10GbE NIC
-
Four cores and 64 GB of memory reserved for a Root Domain with two IB HCAs (SuperCluster M7 only) and 10GbE NICs
-
-
Any other domain in a domain configuration — One core and 16 GB of memory reserved for any remaining Root Domains with one IB HCA and 10GbE NIC
Note - The amount of CPU and memory reserved for Root Domains is sufficient to support only the physical functions in each Root Domain. There is insufficient CPU or memory resources to support zones or applications in Root Domains, so zones and applications are supported only in the I/O Domains.
The remaining CPU and memory resources associated with each Root Domain are parked in CPU and memory repositories, as shown in this diagram.
CPU and memory repositories contain resources not only from the Root Domains, but also any parked resources from the dedicated domains. Whether CPU and memory resources originated from dedicated domains or from Root Domains, once those resources have been parked in the CPU and memory repositories, those resources are no longer associated with their originating domain. These resources become available to I/O Domains.
In addition, CPU and memory repositories contain parked resources only from the compute server that contains the domains providing those parked resources. In other words, if you have two compute servers and both compute servers have Root Domains, there are two sets of CPU and memory repositories, where each compute server has its own CPU and memory repositories with parked resources.
For example, assume you have four domains on your compute server, with three of the four domains as Root Domains, as shown in the previous diagram. Assume each domain has the following IB HCAs and 10GbE NICs, and the following CPU and memory resources:
-
One IB HCA and one 10GbE NIC
-
32 cores
-
512 GB of memory
In this situation, the following CPU and memory resources are reserved for each Root Domain, with the remaining resources available for the CPU and memory repositories:
-
Two cores and 32 GB of memory reserved for the last Root Domains in this configuration. 30 cores and 480 GB of memory available from this Root Domain for the CPU and memory repositories.
-
One core and 16 GB of memory reserved for the second and third Root Domains in this configuration.
-
31 cores and 496 GB of memory available from each of these Root Domains for the CPU and memory repositories.
-
A total of 62 cores (31 x 2) and 992 GB of memory (496 GB x 2) available for the CPU and memory repositories from these two Root Domains.
-
A total of 92 cores (30 + 62 cores) are therefore parked in the CPU repository, and 1472 GB of memory (480 + 992 GB of memory) are parked in the memory repository and are available for the I/O Domains.
With Root Domains, connections to the three networks (client access, IB, and management) go through the physical ports on NIC and HCA, just as they did with dedicated domains. However, the 10GbE NICs and IB HCAs used with Root Domains must also be SR-IOV compliant. SR-IOV compliant cards enable VFs to be created on each card, where the virtualization occurs in the card itself. VFs are not created on the 1GbE NIC for the management network.
The VFs from each Root Domain are parked in the IB VF and 10GbE VF repositories, similar to the CPU and memory repositories, as shown in the following graphic.
Even though the VFs from each Root Domain are parked in the VF repositories, the VFs are created on each 10GbE NIC and IB HCA, so those VFs are associated with the Root Domain that contains those specific 10GbE NIC and IB HCA cards. For example, looking at the example configuration in the previous graphic, the VFs created on the last (rightmost) 10GbE NIC and IB HCA are associated with the last Root Domain.