SuperCluster compute server CPU and memory resources are initially allocated during installation. This base configuration of CPU and memory resources are assigned to I/O Domains in the same proportion as IB HCAs. Memory is assigned in the same proportions. If your computing needs change, you can modify the base configuration later.
You can use the osc-setcoremem command to migrate CPU cores and memory resources between dedicated domains, and from dedicated domains to CPU and memory repositories for the use of I/O domains.
Starting with the 2.5 release, osc-setcoremem makes an attempt to eliminate domain reboots whenever possible. The assessment is transparent. However, if the tool determines that any domain must be restarted as part of a domain reconfiguration session, those domains are highlighted.
You can save time by using the osc-setcoremem simulator to view different possible configurations before you make the changes. The osc-setcoremem simulator uses the same production binary as the osc-setcoremem command. The osc-setcoremem simulator disregards the physical hardware it is running on as long as the target system contains SPARC processors that run the Oracle Solaris OS 11 or later. For instructions on using the simulator, see Run a Simulation.
Use the simulator to test these situations:
Move from a base configuration to a desired configuration.
Move from your current custom configuration to a desired configuration.
Consider this information when you use the osc-setcoremem command:
The final CPU and memory layout for a dedicated domain is optimized for locality to minimize accesses to non-local resources.
The granularity of CPU and memory migration is 1 core and 16GB.
Empty dedicated domains (domains with zero cores and zero memory) are not supported.
The tool tracks resource allocation and ensures that the selections you make are valid. See Minimum and Maximum Resources (Dedicated Domains).
Affected dedicated domains must be rebooted after any change.
The osc-setcoremem command enables you to change the CPU and memory allocations at these levels of granularity:
Socket granularity – The command automatically allocates each domain a minimum of one socket, then enables you to allocate remaining sockets to the domains. See Change CPU/Memory Allocations (Socket Granularity).
Core granularity – The command automatically allocates each domain a minimum number of cores, then enables you to allocate additional cores in one-core increments. See Change CPU/Memory Allocations (Core Granularity).
If you configure the CPU and memory resources so that some resources are not allocated to any domain, those unallocated resources are parked. Parked resources are placed in a logical CPU and memory repository and are available for I/O Domains. See Park Cores and Memory.
Prior to SuperCluster software version 3.0, if you parked resources from dedicated domains, you could not move parked resources to back to dedicated domains after I/O Domains were created. As of version 3.0, you can move parked unallocated resources back to dedicated domains.
The osc-setcoremem command optimizes for locality. For example, in SuperCluster M7, each socket is populated with 32 cores and 512 GB memory. These resources form a locality group, because IO devices, cores, and memory are all local to the same system board. To minimize the impact of NUMA, osc-setcoremem assigns cores and memory from the same socket or locality group. In doing so, the tool enforces a minimum of one memory granule (16 GB memory) for every 4 cores assigned (rounded up) from the same socket. For instance, if a domain was assigned 14 cores from a socket, a minimum of 4 memory granules (14/4=3.5 rounded up to 4) or (4 * 16 GB) = 64 GB memory is required to be allocated from the same socket the 14 cores were allocated from.
When 32 or fewer cores are assigned to a dedicated domain, to optimize performance, those cores are all assigned from a single locality group. Because osc-setcoremem ensures that resources assigned from any locality group must include both cores and memory, memory is assigned from the same locality group as the cores, and is therefore limited to a maximum of 512 GB memory (or less than 512 GB, if fewer than 32 cores are assigned, because some memory must accompany the cores that are assigned to a different domain).
Dedicated domain memory is not limited to 512 GB, though. More than 512 GB of memory can be assigned to a dedicated domain by adding more than 32 cores to that domain. Because the cores, out of necessity, span multiple locality groups, memory can be drawn from multiple locality groups as well. So a domain with 64 cores can include up to 1024 GB of memory, for example, and a domain with 96 cores up to 1536 GB of memory.
Also see Supported Domain Configurations.