I/O Domains

An I/O Domain is an SR-IOV domain that owns its own VFs, each of which is a virtual device based on a PF in one of the Root Domains. Root domains function solely as a provider of VFs to the I/O Domains, based on the physical I/O devices associated with each Root Domain. Applications and zones are supported only in I/O Domains, not in Root Domains.

You can create multiple I/O Domains using the I/O Domain Creation tool. As part of the domain creation process, you also associate one of the following SuperCluster-specific domain types to each I/O Domain:

• Application Domain running Oracle Solaris 11

• Database Domain

Note that only Database Domains that are dedicated domains can host database zones. Database I/O Domains cannot host database zones.

The CPU cores and memory resources owned by an I/O Domain are assigned from the CPU and memory repositories (the cores and memory released from Root Domains on the system) when an I/O Domain is created, as shown in the following graphic.

You use the I/O Domain Creation tool to assign the CPU core and memory resources to the I/O Domains, based on the amount of CPU core and memory resources that you want to assign to each I/O Domain and the total amount of CPU core and memory resources available in the CPU and memory repositories.

Similarly, the IB VFs and 10GbE VFs owned by the I/O Domains come from the IB VF and 10GbE VF repositories (the IB VFs and 10GbE VFs released from Root Domains on the system), as shown in the following graphic.

Again, you use the I/O Domain Creation tool to assign the IB VFs and 10GbE VFs to the I/O Domains using the resources available in the IB VF and 10GbE VF repositories. However, because VFs are created on each 10GbE NIC and IB HCA, the VFs assigned to an I/O Domain will always come from the specific Root Domain that is associated with the 10GbE NIC and IB HCA cards that contain those VFs.

The number and size of the I/O Domains that you can create depends on several factors, including the amount of CPU core and memory resources that are available in the CPU and memory repositories and the amount of CPU core and memory resources that you want to assign to each I/O Domain. However, while it is useful to know the total amount of resources are that are parked in the repositories, it does not necessarily translate into the maximum number of I/O Domains that you can create for your system. In addition, you should not create an I/O Domain that uses more than one socket's worth of resources.

For example, assume that you have 44 cores parked in the CPU repository and 704 GB of memory parked in the memory repository. You could therefore create I/O Domains in any of the following ways:

• One or more large I/O Domains, with each large I/O Domain using one socket's worth of resources (for example, 16 cores and 256 GB of memory)

• One or more medium I/O Domains, with each medium I/O Domain using four cores and 64 GB of memory

• One or more small I/O Domains, with each small I/O Domain using one core and 16 GB of memory

When you go through the process of creating I/O Domains, at some point, the I/O Domain Creation tool will inform you that you cannot create additional I/O Domains. This could be due to several factors, such as reaching the limit of total CPU core and memory resources in the CPU and memory repositories, reaching the limit of resources available specifically to you as a user, or reaching the limit on the number of I/O Domains allowable for this system.

As an example configuration showing how you might assign CPU and memory resources to each domain, assume that you have a domain configuration where one of the domains is a Root Domain, and the other three domains are dedicated domains, as shown in the following figure.

Even though dedicated domains and Root Domains are all shown as equal-sized domains in the preceding figure, that does not mean that CPU core and memory resources must be split evenly across all four domains (where each domain would get 25% of the CPU core and memory resources). Using information that you provide in the configuration worksheets, you can request different sizes of CPU core and memory resources for each domain when your Oracle SuperCluster T5-8 is initially installed.

For example, you could request that each dedicated domain have 30% of the CPU core and memory resources (for a total of 90% of the CPU cores and memory resources allocated to the three dedicated domains), and the remaining 10% allocated to the single Root Domain. Having this configuration would mean that only 10% of the CPU core and memory resources are available for I/O Domains to pull from the CPU and memory repositories. However, you could also request that some of the resources from the dedicated domains be parked at the time of the initial installation of your system, which would further increase the amount of CPU core and memory resources available for I/O Domains to pull from the repositories.

You could also use the CPU/Memory tool after the initial installation to resize the amount of CPU core and memory resources used by the existing domains, depending on the configuration that you chose at the time of your initial installation:

• If all of the domains on your compute server are dedicated domains, you can use the CPU/Memory tool to resize the amount of CPU core and memory resources used by those domains. However, you must reboot those resized dedicated domains if you change the amount of resources using the CPU/Memory tool.

• If you have a mixture of dedicated domains and Root Domains on your compute server:

  • For the dedicated domains, you can use the CPU/Memory tool to resize the amount of CPU core and memory resources used by those dedicated domains. You can also use the tool to park some of the CPU core and memory resources from the dedicated domains, which would park those resources in the CPU and Memory repositories, making them available for the I/O Domains. However, you must reboot those resized dedicated domains if you change the amount of resources using the CPU/Memory tool.

  • For the Root Domains, you cannot resize the amount of CPU core and memory resources for any of the Root Domains after the initial installation. Whatever resources that you asked to have assigned to the Root Domains at the time of initial installation are set and cannot be changed unless you have the Oracle installer come back out to your site to reconfigure your system.

See Configuring CPU and Memory Resources (osc-setcoremem) for more information.

Assume you have a mixture of dedicated domains and Root Domains as mentioned earlier, where each dedicated domain has 30% of the CPU core and memory resources (total of 90% resources allocated to dedicated domains), and the remaining 10% allocated to the Root Domain. You could then make the following changes to the resource allocation, depending on your situation:

• If you are satisfied with the amount of CPU core and memory resources allocated to the Root Domain, but you find that one dedicated domain needs more resources while another needs less, you could reallocate the resources between the three dedicated domains (for example, having 40% for the first dedicated domain, 30% for the second, and 20% for the third), as long as the total amount of resources add up to the total amount available for all the dedicated domains (in this case, 90% of the resources).

• If you find that the amount of CPU core and memory resources allocated to the Root Domain is insufficient, you could park resources from the dedicated domains, which would park those resources in the CPU and Memory repositories, making them available for the I/O Domains. For example, if you find that you need 20% of the resources for I/O Domains created through the Root Domain, you could park 10% of the resources from one or more of the dedicated domains, which would increase the amount of resources in the CPU and Memory repositories by that amount for the I/O Domains.