SuperCluster compute server CPU and memory resources are initially allocated during installation as defined by your configuration. CPU sockets are assigned to domains in the same proportion as IB HCAs. Memory is assigned in the same proportions.
The osc-setcoremem tool enables you to migrate CPU cores and memory resources between dedicated domains, and from dedicated domains to CPU and memory repositories for the use of IO domains.
These points provide important information related to the use of osc-setcoremem:
The final CPU and memory layout for a dedicated domain is optimized for locality to minimize accesses to non-local resources.
The granularity of CPU and memory migration is 1 core and 16GB.
Empty dedicated domains (domains with zero cores and zero memory) are not supported.
The tool tracks resource allocation and ensures that the selections you make are valid. See Minimum and Maximum Resources (Dedicated Domains).
Affected dedicated domains must be rebooted after any change.
The tool enables you to change the CPU and memory allocations in one of two levels of granularity:
Socket granularity – The tool automatically allocates each domain a minimum of one socket, then enables you to allocate remaining sockets to the domains. See Change CPU/Memory Allocations (Socket Granularity).
Core granularity – The tool automatically allocates each domain a minimum number of cores, then enables you to allocate additional cores in one-core increments. See Change CPU/Memory Allocations (Core Granularity).
If you configure the CPU and memory resources so that some resources are not allocated to any domain, those unallocated resources are parked. Parked resources are placed in a logical CPU and memory repository and are available for I/O Domains. See Park Cores and Memory.
You can park resources from dedicated domains anytime, but you cannot move parked resources to dedicated domains once I/O Domains are created.
For SuperCluster M7, each socket is populated with 32 cores and 512 GB memory. These resources form a locality group, because IO devices, cores, and memory are all local to the same system board. To minimize the impact of NUMA, osc-setcoremem assigns cores and memory from the same socket or locality group. In doing so, the tool enforces a minimum of one memory granule (16 GB memory) for every 4 cores assigned (rounded up) from the same socket. For instance, if a domain was assigned 14 cores from a socket, a minimum of 4 memory granules (14/4=3.5 rounded up to 4) or (4 * 16 GB) = 64 GB memory is required to be allocated from the same socket the 14 cores were allocated from.
When 32 or fewer cores are assigned to a dedicated domain, to optimize performance, those cores are all assigned from a single locality group. Because osc-setcoremem ensures that resources assigned from any locality group must include both cores and memory, memory is assigned from the same locality group as the cores, and is therefore limited to a maximum of 512 GB memory (or less than 512 GB, if fewer than 32 cores are assigned, because some memory must accompany the cores that are assigned to a different domain).
Dedicated domain memory is not limited to 512 GB, though. More than 512 GB of memory can be assigned to a dedicated domain by adding more than 32 cores to that domain. Because the cores, out of necessity, span multiple locality groups, memory can be drawn from multiple locality groups as well. So a domain with 64 cores can include up to 1024 GB of memory, for example, and a domain with 96 cores up to 1536 GB of memory.
Also see Supported Domain Configurations.