Using Memory Dynamic Reconfiguration

The Oracle VM Server for SPARC 2.0 release introduces memory dynamic reconfiguration (DR). This feature is capacity-based and enables you to add or remove an arbitrary amount of memory to or from an active logical domain.

The following are the requirements and restrictions for using the memory DR feature:

You can perform memory DR operations on any domain. However, only a single memory DR operation can be in progress on a domain at a given time.
The memory DR feature enforces 256-Mbyte alignment on the address and size of the memory involved in a given operation. See Memory Alignment.
Unaligned memory in the free memory pool cannot be assigned to a domain by using the memory DR feature. See Adding Unaligned Memory.

If the memory of a domain cannot be reconfigured by using a memory DR operation, the domain must be stopped before the memory can be reconfigured. If the domain is the control domain, you must first initiate a delayed reconfiguration.

Adding Memory

If a domain is active, you can use the ldm add-memory command to dynamically add memory to the domain. The ldm set-memory command can also dynamically add memory if the specified memory size is greater than the current memory size of the domain.

Removing Memory

If a domain is active, you can use the ldm remove-memory command to dynamically remove memory from the domain. The ldm set-memory command can also dynamically remove memory if the specified memory size is smaller than the current memory size of the domain.

Memory removal can be a long-running operation. You can track the progress of an operation or cancel an ongoing memory DR request.

Tracking the Progress of a Memory DR Request

You can track the progress of an ldm remove-memory command by running the ldm list -l command for the specified domain.

Canceling a Memory DR Request

You can cancel a removal request that is in progress by interrupting the ldm remove-memory command (by pressing Control-C) or by issuing the ldm cancel-operation memdr command. If you cancel a memory removal request, only the outstanding portion of the removal request is affected, namely, the amount of memory still to be removed from the domain.

Partial Memory DR Requests

A request to dynamically add memory to, or remove memory from, a domain might only be partially fulfilled. This result depends on the availability of suitable memory to add or remove, respectively.

Note - Memory is cleared after it is removed from a domain and before it is added to another domain.

Memory Reconfiguration of the Control Domain

The memory DR feature can be used to reconfigure the memory of the control domain. If a memory DR request cannot be performed on the control domain, you must first initiate a delayed reconfiguration.

Using memory DR might not be appropriate for removing large amounts of memory from an active domain because memory DR operations might be long running. In particular, during the initial configuration of the system, you should use delayed reconfiguration to decrease the memory in the control domain.

Decrease the Control Domain's Memory

Use a delayed reconfiguration instead of a memory DR to decrease the control domain's memory from an initial factory default configuration. In such a case, the control domain owns all of the host system's memory. The memory DR feature is not well suited for this purpose because an active domain is not guaranteed to add or more typically give up all of the requested memory. Rather, the OS running in that domain makes a best effort to fulfill the request. In addition, memory removal can be a long-running operation. These issues are amplified when large memory operations are involved, as is the case for the initial decrease of the control domain's memory.

For these reasons, use a delayed reconfiguration by following these steps:

Use the ldm start-reconf primary command to put the control domain in delayed reconfiguration mode.
Partition the host system's resources that are owned by the control domain, as necessary.
Use the ldm cancel-reconf command to undo the operations in Step 2, if necessary, and start over.
Reboot the control domain to make the reconfiguration changes take effect.

Dynamic and Delayed Reconfiguration

If a delayed reconfiguration is pending in the control domain, a memory reconfiguration request is rejected for any other domain. If a delayed reconfiguration is not pending in the control domain, a memory reconfiguration request is rejected for any domain that does not support memory DR. A memory reconfiguration request on a control domain that does not support memory DR is converted to a delayed reconfiguration request.

Memory Alignment

Memory reconfiguration requests have different alignment requirements that depend on the state of the domain to which the request is applied.

Memory Alignment for Active Domains

Dynamic addition and removal. The address and size of a memory block are 256-Mbyte-aligned for dynamic addition and dynamic removal. The minimum operation size is 256 Mbytes.

A nonaligned request or a removal request that is larger than the bound size is rejected.

Use the following commands to adjust memory allocations:
- ldm add-memory. If you specify the --auto-adj option with this command, the amount of memory to be added is 256-Mbyte-aligned, which might increase the amount of memory actually added to the domain.
- ldm remove-memory. If you specify the --auto-adj option with this command, the amount of memory to be removed is 256-Mbyte-aligned, which might decrease the amount of memory actually removed from the domain.
- ldm set-memory. This command is treated as an addition or a removal operation. If you specify the --auto-adj option, the amount of memory to be added or removed is 256-Mbyte-aligned as previously described. Note that this alignment might increase the resulting memory size of the domain.
Delayed reconfiguration. The address and size of a memory block are 4-Mbyte-aligned. If you make a nonaligned request, the request is rounded up to be 4-Mbyte-aligned.

Memory Alignment for Bound Domains

The address and size of a memory block are 4-Mbyte-aligned for bound domains. If you make a nonaligned request, the request is rounded up to be 4-Mbyte-aligned. This means that the resulting memory size of the domain might be more than you specified.

For the ldm add-memory, ldm set-memory, and ldm remove-memory commands, the --auto-adj option rounds up the size of the resulting memory to be 256-Mbyte-aligned. This means that the resulting memory might be more than you specified.

Memory Alignment for Inactive Domains

For the ldm add-memory, ldm set-memory, and ldm remove-memory commands, the --auto-adj option rounds up the size of the resulting memory to be 256-Mbyte-aligned. There is no alignment requirement for an inactive domain. The restrictions described in Memory Alignment for Bound Domains take effect after such a domain is bound.

Adding Unaligned Memory

The memory DR feature enforces 256-Mbyte memory alignment on the address and size of the memory that is dynamically added to or removed from an active domain. This means that any unaligned memory in an active domain cannot be removed by using memory DR.

This also means than any unaligned memory in the free memory pool cannot be added to an active domain by using memory DR.

After all the aligned memory has been allocated, you can use the ldm add-memory command to add the remaining unaligned memory to a bound or inactive domain. You can also use this command to add the remaining unaligned memory to the control domain by means of a delayed reconfiguration operation.

The following example shows how to add the two remaining 128-Mbyte memory blocks to the primary and ldom1 domains. The ldom1 domain is in the bound state. The following commands add the two remaining memory blocks. The first command initiates a delayed reconfiguration operation on the control domain. The second command adds one of the 128-Mbyte memory blocks to the control domain. The fifth command adds the other 128-Mbyte memory block to the ldom1 domain.

# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the
primary domain reboots, at which time the new configuration for the
primary domain also takes effect.

# ldm add-memory 128M primary
------------------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------

# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -ndcv-  SP      8     2688M    0.1%  23d 8h 8m

# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      8     2560M    0.5%  23d 8h 9m
ldom1            bound      ------  5000    1     524M     

# ldm add-mem 128M ldom1
# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      8     2560M    0.1%  23d 8h 9m
ldom1            bound      ------  5000    1     652M

Memory DR Examples

The following examples show how to perform memory DR operations. For information about the related CLI commands, see the ldm(1M) man page.

Example 10-9 Memory DR Operations on Active Domains

This example shows how to dynamically add memory to and remove it from an active domain, ldom1.

The ldm list output shows the memory for each domain in the Memory field. The first ldm add-mem command exits with an error because you must specify memory in multiples of 256 Mbytes. The next ldm add-mem command uses the --auto-adj option so that even though you specify 200M as the amount of memory to add, the amount is rounded up to 256 Mbytes.

The ldm rm-mem command exits with an error because you must specify memory in multiples of 256 Mbytes. When you add the --auto-adj option to the same command, the memory removal succeeds, as the amount of memory is rounded down to the next 256-Mbyte boundary.

# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      4     27392M   0.4%  1d 22h 53m
ldom1            active     -n----  5000    2     2G       0.4%  1d 1h 23m
ldom2            bound      ------  5001    2     200M          

# ldm add-mem 200M ldom1
The size of memory must be a multiple of 256MB.

# ldm add-mem --auto-adj 200M ldom1
Adjusting request size to 256M.
The ldom1 domain has been allocated 56M more memory
than requested because of memory alignment constraints.

# ldm list
NAME            STATE   FLAGS   CONS    VCPU    MEMORY  UTIL    UPTIME
primary         active  -n-cv-  SP      4       27392M  5.0%    8m
ldom1           active  -n----  5000    2       2304M   0.5%    1m
ldom2           bound   ------  5001    2       200M

# ldm rm-mem --auto-adj 300M ldom1
Adjusting requested size to 256M.
The ldom1 domain has been allocated 44M more memory
than requested because of memory alignment constraints.

# ldm list
NAME            STATE   FLAGS   CONS    VCPU    MEMORY  UTIL    UPTIME
primary         active  -n-cv-  SP      4       27392M  0.3%    8m
ldom1           active  -n----  5000    2       2G      0.2%    2m
ldom2           bound   ------  5001    2       200M

Example 10-10 Memory DR Operations on Bound Domains

This example shows how to add memory to and remove it from a bound domain, ldom2.

The ldm list output shows the memory for each domain in the Memory field. The first ldm add-mem command adds 100 Mbytes of memory to the ldom2 domain. The next ldm add-mem command specifies the --auto-adj option, which causes an additional 112 Mbytes of memory to be dynamically added to ldom2.

The ldm rm-mem command dynamically removes 100 Mbytes from the ldom2 domain. If you specify the --auto-adj option to the same command to remove 300 Mbytes of memory, the amount of memory is rounded down to the next 256-Mbyte boundary.

# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      4     27392M   0.4%  1d 22h 53m
ldom1            active     -n----  5000    2     2G       0.4%  1d 1h 23m
ldom2            bound      ------  5001    2     200M          

# ldm add-mem 100M ldom2

# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      4     27392M   0.5%  1d 22h 54m
ldom1            active     -n----  5000    2     2G       0.2%  1d 1h 25m
ldom2            bound      ------  5001    2     300M       

# ldm add-mem --auto-adj 100M ldom2
Adjusting request size to 256M.
The ldom2 domain has been allocated 112M more memory
than requested because of memory alignment constraints.

# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      4     27392M   0.4%  1d 22h 55m
ldom1            active     -n----  5000    2     2G       0.5%  1d 1h 25m
ldom2            bound      ------  5001    2     512M          

# ldm rm-mem 100M ldom2
# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      4     27392M   3.3%  1d 22h 55m
ldom1            active     -n----  5000    2     2G       0.2%  1d 1h 25m
ldom2            bound      ------  5001    2     412M          

# ldm rm-mem --auto-adj 300M ldom2
Adjusting request size to 256M.
The ldom2 domain has been allocated 144M more memory
than requested because of memory alignment constraints.

# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      4     27392M   0.5%  1d 22h 55m
ldom1            active     -n----  5000    2     2G       0.2%  1d 1h 26m
ldom2            bound      ------  5001    2     256M

Example 10-11 Setting Domain Memory Sizes

This example shows how to use the ldm set-memory command to add memory to and remove it from a domain.

The ldm list output shows the memory for each domain in the Memory field. The first ldm set-mem command attempts to set the primary domain's size to 3400 Mbytes. The resulting error states that the specified value is not on a 256-Mbyte boundary. Adding the --auto-adj option to the same command enables you to successfully remove some memory and stay on the 256-Mbyte boundary. This command also issues a warning to state that not all of the requested memory could be removed as the domain is using that memory.

The next ldm set-mem command sets the memory size of the ldom2 domain, which is in the bound state, to 690 Mbytes. If you add the --auto-adj option to the same command, an additional 78 Mbytes of memory is dynamically added to ldom2 to stay on a 256-Mbyte boundary.

# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      4     27392M   0.5%  1d 22h 55m
ldom1            active     -n----  5000    2     2G       0.2%  1d 1h 26m
ldom2            bound      ------  5001    2     256M

# ldm set-mem 3400M primary
An ldm set-mem 3400M command would remove 23992MB, which is not a multiple
of 256MB. Instead, run ldm rm-mem 23808MB to ensure a 256MB alignment.

# ldm set-mem --auto-adj 3400M primary
Adjusting request size to 3.4G.
The primary domain has been allocated 184M more memory
than requested because of memory alignment constraints.
Only 9472M of memory could be removed from the primary domain
because the rest of the memory is in use.

# ldm set-mem 690M ldom2
# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      4     17920M   0.5%  1d 22h 56m
ldom1            active     -n----  5000    2     2G       0.6%  1d 1h 27m
ldom2            bound      ------  5001    2     690M          

# ldm set-mem --auto-adj 690M ldom2
Adjusting request size to 256M.
The ldom2 domain has been allocated 78M more memory
than requested because of memory alignment constraints.

# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      4     17920M   2.1%  1d 22h 57m
ldom1            active     -n----  5000    2     2G       0.2%  1d 1h 27m
ldom2            bound      ------  5001    2     768M

Skip Navigation Links
Exit Print View
	Oracle VM Server for SPARC 3.0 Administration Guide Oracle VM Server for SPARC