Determining SuperCluster M6-32 Configurations
Determine the Number of Compute Servers
Determine the Number of DCUs in Each Compute Server
Determine the Number of CMUs in Each DCU
Determine the Amount of Memory in Each DCU
Determine the PDomain Configuration on Each Compute Server
Determine the LDom Configuration for Each PDomain
Determining the Best Configuration for Your Situation
Understanding PDomain Configurations
Allocating CPU Resources for LDoms
Allocating Memory Resources for LDoms
Understanding PCIe Cards and Slots for LDoms
Understanding Storage for LDoms
Understanding SuperCluster M6-32
Identifying SuperCluster M6-32 Components
Understanding DCU Configurations
Understanding Half-Populated DCU Root Complexes
Understanding Fully-Populated DCU Root Complexes
Extended Configuration PDomain Overview
Understanding Extended Configuration PDomains
Understanding Base Configuration PDomains
Understanding Compute Server Hardware and Networks
CPU and Memory Resources Overview
LDoms and the PCIe Slots Overview
10GbE Client Access Network Overview
IB Network Data Paths for a Database Domain
IB Network Data Paths for an Application Domain
Understanding SR-IOV Domain Types
Understanding LDom Configurations for Extended Configuration PDomains
Understanding LDom Configurations for Fully-Populated DCUs (Extended Configuration PDomains)
LDom Configurations for Fully-Populated DCUs (Extended Configuration PDomains)
Understanding LDom Configurations for Half-Populated DCUs (Extended Configuration PDomains)
LDom Configurations for Half-Populated DCUs (Extended Configuration PDomains)
Understanding LDom Configurations for Base Configuration PDomains
Understanding LDom Configurations for Fully-Populated DCUs (Base Configuration PDomains)
LDom Configurations for Fully-Populated DCUs (Base Configuration PDomains)
Understanding LDom Configurations for Half-Populated DCUs (Base Configuration PDomains)
LDom Configurations for Half-Populated DCUs (Base Configuration PDomains)
Understanding Clustering Software
Cluster Software for the Database Domain
Cluster Software for the Oracle Solaris Application Domains
Understanding System Administration Resources
Understanding Platform-Specific Oracle ILOM Features
Oracle ILOM Remote Console Plus Overview
Oracle Hardware Management Pack Overview
Time Synchronization and NTP Service
Multidomain Extensions to Oracle ILOM MIBs
Hardware Installation Overview
Hardware Installation Task Overview
Hardware Installation Documents
Preparing the Site (Storage Rack and Expansion Racks)
Prepare the Site for the Racks
Network Infrastructure Requirements
Compute Server Default Host Names and IP Addresses
Compute Server Network Components
Storage Rack Network Components
Cable the ZFS Storage Appliance
ZFS Appliance Power Cord Connection Reference
ZFS Storage Appliance Cabling Reference
Leaf Switch 1 Cabling Reference
Leaf Switch 2 Cabling Reference
IB Switch-to-Switch Cabling Reference
Cable the Ethernet Management Switch
Ethernet Management Switch Cabling Reference
Connect SuperCluster M6-32 to the Facility Networks
Expansion Rack Default IP Addresses
Understanding Internal Cabling (Expansion Rack)
Understanding SuperCluster Software
Identify the Version of SuperCluster Software
Controlling SuperCluster M6-32
Powering Off SuperCluster M6-32 Gracefully
Power Off SuperCluster M6-32 in an Emergency
Monitoring SuperCluster M6-32 (OCM)
Monitoring the System With ASR
Configure ASR on the Compute Servers (Oracle ILOM)
Configure SNMP Trap Destinations for Storage Servers
Configure ASR on the ZFS Storage Appliance
Configuring ASR on the Compute Servers (Oracle Solaris 11)
Approve and Verify ASR Asset Activation
Change ssctuner Properties and Disable Features
Configuring CPU and Memory Resources (osc-setcoremem)
Minimum and Maximum Resources (Dedicated Domains)
Supported Domain Configurations
Plan CPU and Memory Allocations
Display the Current Domain Configuration (osc-setcoremem)
Display the Current Domain Configuration (ldm)
Change CPU/Memory Allocations (Socket Granularity)
Change CPU/Memory Allocations (Core Granularity)
Access osc-setcoremem Log Files
Revert to a Previous CPU/Memory Configuration
Remove a CPU/Memory Configuration
Obtaining the EM Exadata Plug-in
Known Issues With the EM Exadata Plug-in
Configuring the Exalogic Software
Prepare to Configure the Exalogic Software
Enable Domain-Level Enhancements
Enable Cluster-Level Session Replication Enhancements
Configuring Grid Link Data Source for Dept1_Cluster1
Configuring SDP-Enabled JDBC Drivers for Dept1_Cluster1
Create an SDP Listener on the IB Network
Administering Oracle Solaris 11 Boot Environments
Advantages to Maintaining Multiple Boot Environments
Mount to a Different Build Environment
Reboot to the Original Boot Environment
Create a Snapshot of a Boot Environment
Remove Unwanted Boot Environments
Monitor Write-through Caching Mode
An I/O Domain is an SR-IOV domain that owns its own VFs, each of which is a virtual device based on a PF in one of the Root Domains. Root domains function solely as a provider of VFs to the I/O Domains, based on the physical I/O devices associated with each Root Domain. Applications and zones are supported only in I/O Domains, not in Root Domains.
You can create multiple I/O Domains using the I/O Domain Creation tool. As part of the domain creation process, you also associate one of the following SuperCluster-specific domain types to each I/O Domain:
Application Domain running Oracle Solaris 11, or Application I/O Domain
Database I/O Domain
Note that only Database Domains that are dedicated domains can host database zones. Database I/O Domains cannot host database zones.
The CPU cores and memory resources owned by an I/O Domain are assigned from the CPU and memory repositories (the cores and memory released from Root Domains on the system) when an I/O Domain is created, as shown in the following graphic.
You use the I/O Domain Creation tool to assign the CPU core and memory resources to the I/O Domains, based on the amount of CPU core and memory resources that you want to assign to each I/O Domain and the total amount of CPU core and memory resources available in the CPU and memory repositories.
Similarly, the IB VFs and 10GbE VFs owned by the I/O Domains come from the IB VF and 10GbE VF repositories (the IB VFs and 10GbE VFs released from Root Domains on the system), as shown in the following graphic.
Again, you use the I/O Domain Creation tool to assign the IB VFs and 10GbE VFs to the I/O Domains using the resources available in the IB VF and 10GbE VF repositories. However, because VFs are created on each EMS and IB HCA, the VFs assigned to an I/O Domain will always come from the specific Root Domain that is associated with the EMSs and IB HCAs that contain those VFs.
The number and size of the I/O Domains that you can create depends on several factors, including the amount of CPU core and memory resources that are available in the CPU and memory repositories and the amount of CPU core and memory resources that you want to assign to each I/O Domain. However, while it is useful to know the total amount of resources are that are parked in the repositories, it does not necessarily translate into the maximum number of I/O Domains that you can create for your system. In addition, you should not create an I/O Domain that uses more than one socket's worth of resources.
For example, assume that you have 32 cores parked in the CPU repository and 1472 GB of memory parked in the memory repository. You could therefore create I/O Domains in any of the following ways:
One or more large I/O Domains, with each large I/O Domain using one socket's worth of resources (for example, 12 cores and 512 GB of memory)
One or more medium I/O Domains, with each medium I/O Domain using four cores and 64 GB of memory
One or more small I/O Domains, with each small I/O Domain using one core and 16 GB of memory
When you go through the process of creating I/O Domains, at some point, the I/O Domain Creation tool will inform you that you cannot create additional I/O Domains. This could be due to several factors, such as reaching the limit of total CPU core and memory resources in the CPU and memory repositories, reaching the limit of resources available specifically to you as a user, or reaching the limit on the number of I/O Domains allowable for this system.
Note - The following examples describe how resources might be divided up between domains using percentages to make the conceptual information easier to understand. However, you actually divide CPU core and memory resources between domains at a socket granularity or core granularity level. Refer to the Oracle SuperCluster M6-32 Owner's Guide: Administration for more information.
As an example configuration showing how you might assign CPU and memory resources to each domain, assume that you have a domain configuration where one of the domains is a Root Domain, and the other three domains are dedicated domains, as shown in the following figure.
Even though dedicated domains and Root Domains are all shown as equal-sized domains in the preceding figure, that does not mean that CPU core and memory resources must be split evenly across all four domains (where each domain would get 25% of the CPU core and memory resources). Using information that you provide in the configuration worksheets, you can request different sizes of CPU core and memory resources for each domain when your SuperCluster M6-32 is initially installed.
For example, you could request that each dedicated domain have 30% of the CPU core and memory resources (for a total of 90% of the CPU cores and memory resources allocated to the three dedicated domains), and the remaining 10% allocated to the single Root Domain. Having this configuration would mean that only 10% of the CPU core and memory resources are available for I/O Domains to pull from the CPU and memory repositories. However, you could also request that some of the resources from the dedicated domains be parked at the time of the initial installation of your system, which would further increase the amount of CPU core and memory resources available for I/O Domains to pull from the repositories.
You could also use the CPU/Memory tool after the initial installation to resize the amount of CPU core and memory resources used by the existing domains, depending on the configuration that you chose at the time of your initial installation:
If all of the domains on your compute server are dedicated domains, you can use the CPU/Memory tool to resize the amount of CPU core and memory resources used by those domains. However, you must reboot those resized dedicated domains if you change the amount of resources using the CPU/Memory tool.
If you have a mixture of dedicated domains and Root Domains on your compute server:
For the dedicated domains, you can use the CPU/Memory tool to resize the amount of CPU core and memory resources used by those dedicated domains. You can also use the tool to park some of the CPU core and memory resources from the dedicated domains, which would park those resources in the CPU and Memory repositories, making them available for the I/O Domains. However, you must reboot those resized dedicated domains if you change the amount of resources using the CPU/Memory tool.
For the Root Domains, you cannot resize the amount of CPU core and memory resources for any of the Root Domains after the initial installation. Whatever resources that you asked to have assigned to the Root Domains at the time of initial installation are set and cannot be changed unless you have the Oracle installer come back out to your site to reconfigure your system.
Refer to the Oracle SuperCluster M6-32 Owner's Guide: Administration for more information.
Assume you have a mixture of dedicated domains and Root Domains as mentioned earlier, where each dedicated domain has 30% of the CPU core and memory resources (total of 90% resources allocated to dedicated domains), and the remaining 10% allocated to the Root Domain. You could then make the following changes to the resource allocation, depending on your situation:
If you are satisfied with the amount of CPU core and memory resources allocated to the Root Domain, but you find that one dedicated domain needs more resources while another needs less, you could reallocate the resources between the three dedicated domains (for example, having 40% for the first dedicated domain, 30% for the second, and 20% for the third), as long as the total amount of resources add up to the total amount available for all the dedicated domains (in this case, 90% of the resources).
If you find that the amount of CPU core and memory resources allocated to the Root Domain is insufficient, you could park resources from the dedicated domains, which would park those resources in the CPU and Memory repositories, making them available for the I/O Domains. For example, if you find that you need 20% of the resources for I/O Domains created through the Root Domain, you could park 10% of the resources from one or more of the dedicated domains, which would increase the amount of resources in the CPU and Memory repositories by that amount for the I/O Domains.