This section describes general known issues about this release of LDoms software that are broader than a specific bug number. Workarounds are provided where available.
For discussions in Logical Domains documentation, the terms service processor (SP) and system controller (SC) are interchangeable.
The following cards are not supported for this LDoms 1.3 software release:
Sun Dual Port 4x IB Host Channel Adapter PCI-X Card
Dual Port 4x PCI EXPRESS® Infiniband Host Channel Adapter - Low Profile
If these unsupported configurations are used with LDoms 1.3, stop and unbind all logical domains before the control domain is rebooted. Failure to do so can result in a system crash causing the loss of all the logical domains that are active in the system.
If a service domain is running a version of Solaris 10 OS prior to Solaris 10 10/09 and is exporting a physical disk slice as a virtual disk to a guest domain, then this virtual disk will appear in the guest domain with an inappropriate device ID. If that service domain is then upgraded to Solaris 10 10/09, the physical disk slice exported as a virtual disk will appear in the guest domain with no device ID.
This removal of the device ID of the virtual disk can cause problems to applications attempting to reference the device ID of virtual disks. In particular, this can cause the Solaris Volume Manager (SVM) to be unable to find its configuration or to access its metadevices.
Workaround: After upgrading a service domain to Solaris 10 10/09, if a guest domain is unable to find its SVM configuration or its metadevices, execute the following procedure.
Boot the guest domain.
Disable the devid feature of SVM by adding the following lines to the /kernel/dr/md.conf file:
Reboot the guest domain.
After the domain has booted, the SVM configuration and metadevices should be available.
Check the SVM configuration and ensure that it is correct.
Re-enable the SVM devid feature by removing from the /kernel/drv/md.conf file the two lines that you added in Step 2.
Reboot the guest domain.
During the reboot, you will see messages similar to this:
NOTICE: mddb: unable to get devid for 'vdc', 0x10
These messages are normal and do not report any problems.
There is a limit to the number of LDCs available in any logical domain. For UltraSPARC T2 based platforms, that limit is 512. For UltraSPARC T2 Plus based platforms, that limit is 768. This only becomes an issue on the control domain because the control domain has at least part, if not all, of the I/O subsystem allocated to it. This might also be an issue because of the potentially large number of LDCs that are created for both virtual I/O data communications and the Logical Domains Manager control of the other logical domains.
If you try to add a service, or bind a domain, so that the number of LDC channels exceeds the limit on the control domain, the operation fails with an error message similar to the following:
13 additional LDCs are required on guest primary to meet this request, but only 9 LDCs are available
The following guidelines can help prevent creating a configuration that could overflow the LDC capabilities of the control domain:
The control domain allocates 12 LDCs for various communication purposes with the hypervisor, Fault Management Architecture (FMA), and the system controller (SC), independent of the number of other logical domains configured.
The control domain allocates 1 LDC to every logical domain, including itself, for control traffic.
Each virtual I/O service on the control domain consumes 1 LDC for every connected client of that service.
For example, consider a control domain and 8 additional logical domains. Each logical domain needs the following at a minimum:
Applying the above guidelines yields the following results (numbers in parentheses correspond to the preceding guideline number from which the value was derived):
12(1) + 9(2) + 8 x 3(3)=45 LDCs in total.
Now consider the case where there are 45 domains instead of 8, and each domain includes 5 virtual disks, 5 virtual networks, and a virtual console. Now the equation becomes:
12 + 46 + 45 x 11=553 LDCs in total.
Depending upon the number of supported LDCs of your platform, the Logical Domains Manager will either accept or reject the configurations.
Logical Domains software does not impose a memory size limitation when creating a domain. The memory size requirement is a characteristic of the guest operating system. Some Logical Domains functionality might not work if the amount of memory present is less than the recommended size. For recommended and minimum size memory requirements, see the installation guide for the operating system you are using.
OpenSolaris 2009.06 OS. See “System Requirements” in Getting Started With OpenSolaris 2009.06.
The OpenBootTM PROM has a minimum size restriction for a domain. Currently, that restriction is 12 Mbytes. If you have a domain less than that size, the Logical Domains Manager will automatically boost the size of the domain to 12 Mbytes. Refer to the release notes for your system firmware for information about memory size requirements.
You can the boot following number of domains depending on your platform:
Up to 128 on UltraSPARC T2 Plus based servers
Up to 64 on UltraSPARC T2 based servers
If unallocated virtual CPUs are available, assign them to the service domain to help process the virtual I/O requests. Allocate 4 to 8 virtual CPUs to the service domain when creating more than 32 domains. In cases where maximum domain configurations have only a single CPU in the service domain, do not put unnecessary stress on the single CPU when configuring and using the domain. The virtual switch (vsw) services should be spread over all the network adapters available in the machine. For example, if booting 128 domains on a Sun SPARC Enterprise T5240 server, create 4 vsw services, each serving 32 virtual net (vnet) instances. Do not have more than 32 vnet instances per vsw service because having more than that tied to a single vsw could cause hard hangs in the service domain.
To run the maximum configurations, a machine needs the following amount of memory, depending on your platform, so that the guest domains contain an adequate amount of memory:
128 Gbytes of memory for UltraSPARC T2 Plus based servers
64 Gbytes of memory for UltraSPARC T2 based servers
Memory and swap space usage increases in a guest domain when the vsw services used by the domain provides services to many virtual networks (in multiple domains). This is due to the peer-to-peer links between all the vnet connected to the vsw. The service domain benefits from having extra memory. Four Gbytes is the recommended minimum when running more than 64 domains. Start domains in groups of 10 or less and wait for them to boot before starting the next batch. The same advice applies to installing operating systems on domains.
If you have made any configuration changes since last saving a configuration to the SC, before you attempt to power off or power cycle a Logical Domains system, make sure that you save the latest configuration that you want to keep.
Shut down and unbind all the non-I/O domains.
Shut down and unbind any active I/O domains.
Because no other domains are bound, the firmware automatically powers off the system.
Shut down and unbind all the non-I/O domains.
Shut down and unbind any active I/O domains.
Because no other domains are bound, the firmware automatically power cycles the system before rebooting it. When the system restarts, it boots into the Logical Domains configuration last saved or explicitly set.
Under certain circumstances, the Logical Domains Manager rounds up the requested memory allocation to either the next largest 8-Kbyte or 4-Mbyte multiple. This can be seen in the following example output of the ldm list-domain -l command, where the constraint value is smaller than the actual allocated size:
Memory: Constraints: 1965 M raddr paddr5 size 0x1000000 0x291000000 1968M
Variable updates persist across a reboot, but not across a powercycle, unless the variable updates are either initiated from OpenBoot firmware on the control domain or followed by saving the configuration to the SC.
In this context, it is important to note that a reboot of the control domain could initiate a powercycle of the system:
When the control domain reboots, if there are no bound guest domains, and no delayed reconfiguration in progress, the SC powercycles the system.
When the control domain reboots, if there are guest domains bound or active (or the control domain is in the middle of a delayed reconfiguration), the SC does not powercycle the system.
LDom variables for a domain can be specified using any of the following methods:
At the OpenBoot prompt
Using the Solaris OS eeprom(1M) command
Using the Logical Domains Manager CLI (
Modifying, in a limited fashion, from the system controller (SC) using the bootmode command, that is, only certain variables, and only when in the factory-default configuration
The goal is that, variable updates that are made by using any of these methods always persist across reboots of the domain. The variable updates also always reflect in any subsequent logical domain configurations that were saved to the SC.
In LDoms 1.3 software, there are a few cases where variable updates do not persist as expected:
All methods of updating a variable persist across reboots of that domain. However, they do not persist across a powercycle of the system, unless a subsequent logical domain configuration is saved to the SC. The methods of updating a variable include by OpenBoot firmware and by the eeprom and ldm commands. In addition, in the control domain, updates made using OpenBoot firmware persist across a powercycle of the system, that is, even without subsequently saving a new logical domain configuration to the SC.
In all cases, when reverting to the factory-default configuration from a configuration generated by the Logical Domains Manager, all LDoms variables start with their default values.
If you are concerned about Logical Domains variable changes, do one of the following:
Bring the system to the ok prompt and update the variables.
Update the variables while the Logical Domains Manager is disabled:
# svcadm disable ldmd update variables # svcadm enable ldmd
When running Live Upgrade, perform the following steps:
# svcadm disable -t ldmd # luactivate be3 # init 6
The following Bug IDs have been filed to resolve these issues: 6520041, 6540368, 6540937, and 6590259.
Sun Simple Management Network Protocol (SNMP) Management Agent does not support multiple domains. Only a single global domain is supported.
Using CPU dynamic reconfiguration (DR) to power down virtual CPUs does not work with processor sets, resource pools, or the zone's dedicated CPU feature.
When using CPU power management in elastic mode, the Solaris OS guest sees only the CPUs that are allocated to the domains that are powered on. That means that output from the psrinfo(1M) command dynamically changes depending on the number of CPUs currently power-managed. This causes an issue with processor sets and pools, which require actual CPU IDs to be static to allow allocation to their sets. This can also impact the zone's dedicated CPU feature.
Workaround: Set the performance mode for the power management policy.
There are several issues associated with FMA and power-managing CPUs. If a CPU faults when running in elastic mode, switch to performance mode until the faulted CPU recovers. If all faulted CPUs recover, then elastic mode can be used again.
For more information about faulted resources, see the OpenSolaris Fault Management web page.
When a primary domain is in a delayed reconfiguration state, CPUs are power managed only after the primary domain reboots. This means that CPU power management will not bring additional CPUs online when the domain is experiencing high-load usage until the primary domain reboots, clearing the delayed reconfiguration state.
Domain migrations are not supported for a source or target machine in elastic mode. If a migration is underway while in performance mode and the power management policy is set to elastic mode, the policy switch is deferred until the migration completes. The migration command returns an error if either the source or target machine is in elastic mode and a domain migration is attempted.
The Solaris 10 10/09 OS introduces the capability to dynamically add and remove cryptographic units from a domain, which is called cryptographic unit dynamic reconfiguration (DR). The Logical Domains Manager automatically detects whether a domain allows cryptographic unit DR, and enables the functionality only for those domains. In addition, CPU DR is no longer disabled in domains that have cryptographic units bound and are running an appropriate version of the Solaris OS.
However, the restriction against enabling power management (PM) elastic mode still applies if there are any domains with cryptographic units bound. This restriction is enforced by the Logical Domains Manager. So, any attempt to change power management's policy to elastic from the SP when there are cryptographic units bound to any domain would be ineffective and is therefore not recommended.