Perform this procedure on each compute node to move CPU and memory resources from dedicated domains into logical CPU and memory repositories, making the resources available for I/O Domains.
If you are parking cores and memory, plan carefully. Once you park resources and create I/O Domains you cannot move resources back to dedicated domains.
The examples in this procedure show a SuperCluster T5-8 full rack. The concepts in this procedure also apply to other SuperCluster models.
This is an example of a plan that reduces resources for primary, ssccn2-dom2, ssccn2-dom4, so that the resources are parked (unallocated). The parked resources can later be used by I/O Domains.
This table shows the allocation plan (see Plan CPU and Memory Allocations).
|
The tool does not continue if any inactive domains are present.
In this example, some resources are left unallocated which parks them.
Respond when prompted. Press Enter to select the default value.
# /opt/oracle.supercluster/bin/osc-setcoremem
osc-setcoremem
v1.0 built on Oct 29 2014 10:21:05
Current Configuration: Full-Rack T5-8 SuperCluster
+-------------------------+-------+--------+-----------+--- MINIMUM ----+
| DOMAIN | CORES | MEM_GB | TYPE | CORES | MEM_GB |
+-------------------------+-------+--------+-----------+-------+--------+
| primary | 32 | 512 | Dedicated | 9 | 64 |
| ssccn2-dom1 | 16 | 256 | Dedicated | 4 | 32 |
| ssccn2-dom2 | 16 | 256 | Dedicated | 4 | 32 |
| ssccn2-dom3 | 16 | 256 | Dedicated | 4 | 32 |
| ssccn2-dom4 | 16 | 256 | Dedicated | 4 | 32 |
| ssccn2-dom5 | 4 | 64 | Root | 4 | 64 |
+-------------------------+-------+--------+-----------+-------+--------+
| unallocated or parked | 28 | 448 | -- | -- | -- |
+-------------------------+-------+--------+-----------+-------+--------+
[Note] Following domains will be skipped in this session.
Root Domains
------------
ssccn2-dom5
CPU allocation preference:
1. Socket level
2. Core level
In case of Socket level granularity, proportional memory capacity is
automatically selected for you.
Choose Socket or Core level [S or C] C
Step 1 of 2: Core count selection
primary : desired number of cores [min: 9, max: 80. default: 32] : 16
you chose [16] cores for primary domain
ssccn2-dom1 : desired number of cores [min: 4, max: 68. default: 16] : <Enter>
you chose [16] cores for ssccn2-dom1 domain
ssccn2-dom2 : desired number of cores [min: 4, max: 56. default: 16] : 8
you chose [8] cores for ssccn2-dom2 domain
ssccn2-dom3 : desired number of cores [min: 4, max: 52. default: 16] : <Enter>
you chose [16] cores for ssccn2-dom3 domain
ssccn2-dom4 : desired number of cores [min: 4, max: 40. default: 16] : 4
you chose [4] cores for ssccn2-dom4 domain
New Configuration in progress after Core count selection:
+-------------------------+-------+--------+-----------+--- MINIMUM ----+
| DOMAIN | CORES | MEM_GB | TYPE | CORES | MEM_GB |
+-------------------------+-------+--------+-----------+-------+--------+
| primary | 16 | 512 | Dedicated | 9 | 64 |
| ssccn2-dom1 | 16 | 256 | Dedicated | 4 | 64 |
| ssccn2-dom2 | 8 | 256 | Dedicated | 4 | 32 |
| ssccn2-dom3 | 16 | 256 | Dedicated | 4 | 64 |
| ssccn2-dom4 | 4 | 256 | Dedicated | 4 | 32 |
| *ssccn2-dom5 | 4 | 64 | Root | 4 | 64 |
+-------------------------+-------+--------+-----------+-------+--------+
| unallocated or parked | 64 | 448 | -- | -- | -- |
+-------------------------+-------+--------+-----------+-------+--------+
Step 2 of 2: Memory selection
primary : desired memory capacity in GB (must be 16 GB aligned) [min: 64, max: 1344. default: 256] : <Enter>
you chose [256 GB] memory for primary domain
ssccn2-dom1 : desired memory capacity in GB (must be 16 GB aligned) [min: 64, max: 1152. default: 256] : <Enter>
you chose [256 GB] memory for ssccn2-dom1 domain
ssccn2-dom2 : desired memory capacity in GB (must be 16 GB aligned) [min: 32, max: 928. default: 128] : 64
you chose [64 GB] memory for ssccn2-dom2 domain
ssccn2-dom3 : desired memory capacity in GB (must be 16 GB aligned) [min: 64, max: 928. default: 256] : <Enter>
you chose [256 GB] memory for ssccn2-dom3 domain
ssccn2-dom4 : desired memory capacity in GB (must be 16 GB aligned) [min: 32, max: 704. default: 64] : <Enter>
you chose [64 GB] memory for ssccn2-dom4 domain
New Configuration in progress after Memory selection:
+-------------------------+-------+--------+-----------+--- MINIMUM ----+
| DOMAIN | CORES | MEM_GB | TYPE | CORES | MEM_GB |
+-------------------------+-------+--------+-----------+-------+--------+
| primary | 16 | 256 | Dedicated | 9 | 64 |
| ssccn2-dom1 | 16 | 256 | Dedicated | 4 | 64 |
| ssccn2-dom2 | 8 | 64 | Dedicated | 4 | 32 |
| ssccn2-dom3 | 16 | 256 | Dedicated | 4 | 64 |
| ssccn2-dom4 | 4 | 64 | Dedicated | 4 | 32 |
| *ssccn2-dom5 | 4 | 64 | Root | 4 | 64 |
+-------------------------+-------+--------+-----------+-------+--------+
| unallocated or parked | 64 | 1088 | -- | -- | -- |
+-------------------------+-------+--------+-----------+-------+--------+
The following domains will be stopped and restarted:
ssccn2-dom4
ssccn2-dom2
This configuration requires rebooting the control domain.
Do you want to proceed? Y/N : Y
IMPORTANT NOTE:
+----------------------------------------------------------------------------+
|After the reboot, osc-setcoremem attempts to complete CPU, memory re-configuration|
| Please check syslog and the state of all domains before using the system. |
| eg., dmesg | grep osc-setcoremem ; ldm list | grep -v active ; date |
+----------------------------------------------------------------------------+
All activity is being recorded in log file:
/opt/oracle.supercluster/osc-setcoremem/log/osc-setcoremem_activity_10-29-2014_17:29:41.log
Please wait while osc-setcoremem is setting up the new CPU, memory configuration.
It may take a while. Please be patient and do not interrupt.
0% 10 20 30 40 50 60 70 80 90 100%
|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|
*=====*=====*=====*=====*=====*=====*=====*=====*=====*=====*
Broadcast Message from root (pts/1) on etc27dbadm0201 Wed Oct 29 17:31:48...
THE SYSTEM etc27dbadm0201 IS BEING SHUT DOWN NOW ! ! !
Log off now or risk your files being damaged
Task complete with no errors.
When the system is back online, check the log file to ensure that all re-configuration steps were successful.
# tail -20 /opt/oracle.supercluster/osc-setcoremem/log/osc-setcoremem_activity_10-29-2014_17:29:41.log
Task complete with no errors.
::Post-reboot activity::
Please wait while osc-setcoremem is setting up the new CPU, memory configuration.
It may take a while. Please be patient and do not interrupt.
Executing ldm commands ..
0% 10 20 30 40 50 60 70 80 90 100%
|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|
*=====*=====*=====*=====*=====*=====*=====*=====*=====*=====*
Task complete with no errors.
This concludes socket/core, memory reconfiguration.
You can continue using the system.
You can verify the resource allocation and check for possible osc-setcoremem errors in several ways:
In this example, there are 64 parked cores from these domains:
28 cores were parked prior to the resource allocation (These cores were placed in the logical CPU repository when the Root Domain was created)
36 cores are parked from ssccn2-dom1, ssccn2-dom3, and ssccn2-dom5.
# ldm list-devices -p core | grep cid | wc -l
64
In this example, memory is parked:
2 x 224 memory blocks are parked from the Root Domain.
The other memory blocks are parked from the dedicated domains.
When the memory is added together it equals approximately 1088 GB.
# ldm list-devices memory
MEMORY
PA SIZE
0x30000000 15616M
0x2400000000 114176M
0x82000000000 128G
0x181000000000 192G
0x281000000000 192G
0x300000000000 224G
0x380000000000 224G