Recovering Domains After Detecting Faulty or Missing Resources
If a server starting with the SPARC T5, SPARC M5, SPARC S7 series server, or the Fujitsu M10 server detects a faulty or missing resource at power on, the Logical Domains Manager attempts to recover the configured domains by using the remaining available resources. While the recovery takes place, the system (or physical domain on SPARC M-series servers) is said to be in recovery mode. Recovery mode is enabled by default. See Controlling Recovery Mode.
The I/O topology within each PCIe switch in the configuration does not match the I/O topology of the last selected power-on configuration
CPU resources or memory resources of the last selected power-on configuration are no longer present in the system
At power on, the system firmware reverts to the factory-default configuration if the last selected power-on configuration cannot be booted in any of the following circumstances:
When recovery mode is enabled, the Logical Domains Manager recovers all active and bound domains from the last selected power-on configuration. The resulting running configuration is called the degraded configuration. The degraded configuration is saved to the SP and remains the active configuration until either a new configuration is saved or the physical domain is power-cycled.
Note - The physical domain does not require a power cycle to activate the degraded configuration after recovery as it is already the running configuration.
If the physical domain is power-cycled, the system firmware first attempts to boot the last original power-on configuration. That way, if the missing or faulty hardware was replaced in the meantime, the system can boot the original normal configuration. If the last selected power-on configuration is not bootable, the firmware next attempts to boot the associated degraded configuration if it exists. If the degraded configuration is not bootable or does not exist, the factory-default configuration is booted and recovery mode is invoked.
Control domain. The Logical Domains Manager recovers the control domain by restoring its CPU, memory, and I/O configuration as well as its virtual I/O services.
If the amount of CPU or memory required for all recoverable domains is larger than the remaining available amounts, the number of CPUs or cores or memory is reduced in proportion to the size of the other domains. For example, in a four-domain system where each domain has 25% of the CPUs and memory assigned, the resulting degraded configuration still assigns 25% of the CPUs and memory to each domain. If the primary domain originally had up to two cores (16 virtual CPUs) and eight Gbytes of memory, the control domain size is not reduced.
Root complexes and PCIe devices that are assigned to other domains are removed from the control domain. The virtual functions on root complexes that are owned by the control domain are re-created. Any missing root complex, PCIe device, physical function, or virtual function that is assigned to the control domain is marked as evacuated. The Logical Domains Manager then reboots the control domain to make the changes active.
Root domains. After the control domain has been rebooted, the Logical Domains Manager recovers the root domains. The amount of CPU and memory is reduced in proportion to the other recoverable domains, if needed. If a root complex is no longer physically present in the system, it is marked as evacuated. This root complex is not configured into the domain during the recovery operation. A root domain is recovered as long as at least one of the root complexes that is assigned to the root domain is available. If none of its root complexes are available, the root domain is not recovered. The Logical Domains Manager boots the root domain and re-creates the virtual functions on the physical functions that are owned by the root domain. It also removes the PCIe slots that are loaned out by the root domain. Any missing PCIe slots, physical functions, and virtual functions are marked as evacuated. Any virtual I/O services that are provided by the domain are re-created, if possible.
I/O domains. Logical Domains Manager recovers any I/O domains. Any PCIe slots and virtual functions that are missing from the system are marked as evacuated. If none of the required I/O devices are present, the domain is not recovered and its CPU and memory resources are available for use by other domains. Any virtual I/O services that are provided by the domain are re-created, if possible.
Guest domains. A guest domain is recovered only if at least one of the service domains that serves the domain has been recovered. If the guest domain cannot be recovered, its CPU and memory resources are available for use by other guest domains.
The recovery operation works in the following order:
When possible, the same number of CPUs and amount of memory are allocated to a domain as specified by the original configuration. If that number of CPUs or amount of memory are not available, these resources are reduced proportionally to consume the remaining available resources. If you assigned named resources to a domain and it is later recovered in recovery mode, there is no attempt to re-assign those named resources to that domain.
Note - When a system is in recovery mode, you can only perform ldm list-* commands. All other ldm commands are disabled until the recovery operation completes.
The Logical Domains Manager only attempts to recover bound and active domains. The existing resource configuration of any unbound domain is copied to the new configuration as-is.
During a recovery operation, fewer resources might be available than in the previously booted configuration. As a result, the Logical Domains Manager might only be able to recover some of the previously configured domains. Also, a recovered domain might not include all of the resources from its original configuration. For example, a recovered bound domain might have fewer I/O resources than it had in its previous configuration. A domain might not be recovered if its I/O devices are no longer present or if its parent service domain could not be recovered.
Recovery mode records its steps to the Logical Domains Manager SMF log, /var/svc/log/ldoms-ldmd:default.log. A message is written to the system console when Logical Domains Manager starts a recovery, reboots the control domain, and when the recovery completes.
 | Caution - A recovered domain is not guaranteed to be completely operable. The domain might not include a resource that is essential to run the OS instance or an application. For example, a recovered domain might only have a network resource and no disk resource. Or, a recovered domain might be missing a file system that is required to run an application. Using multipathed I/O for a domain reduces the impact of missing I/O resources. |
Recovery Mode Hardware and Software Requirements
Hardware Requirements – The recovery mode feature is supported on servers starting with the SPARC T5, SPARC M5, and SPARC S7 series server and the Fujitsu M10 server.
Firmware Requirements – At least version 9.1.0.a of the system firmware for the SPARC T5, SPARC M5, and SPARC M6 server. At least version 9.4.3 of the system firmware for the SPARC T7 and SPARC M7 series server. Any released version of the system firmware for the SPARC S7, SPARC T8, and SPARC M8 series server. At least version XCP2230 of the system firmware for the Fujitsu M10 server. At least version XCP3021 of the system firmware for the Fujitsu SPARC M12 server.
Software Requirements – Non-primary root domains that loan out PCIe slots must be running at least the Oracle Solaris 10 1/13 OS or the Oracle Solaris 11.2 OS.
Degraded Configuration
Each physical domain can have only one degraded configuration saved to the SP. If a degraded configuration already exists, it is replaced by the newly created degraded configuration.
You cannot interact directly with degraded configurations. The system firmware transparently boots the degraded version of the next power-on configuration, if necessary. This transparency enables the system to boot the original configuration after a power cycle when the missing resources reappear. When the active configuration is a degraded configuration, it is marked as [degraded] in the ldm list-spconfig output.
The autosave functionality is disabled while the active configuration is a degraded configuration. If you save a new configuration to the SP while a degraded configuration is active, the new configuration is a normal non-degraded configuration.
Note - A previously missing resource that reappears on a subsequent power cycle has no effect on the contents of a normal configuration. However, if you subsequently select the configuration that triggered recovery mode, the SP boots the original, non-degraded configuration now that all its hardware is available.
Controlling Recovery Mode
The ldmd/recovery_mode SMF property controls recovery mode behavior. Recovery mode is enabled by default.
When the ldmd/recovery_mode property is not present or is set to auto, recovery mode is enabled.
When the ldmd/recovery_mode property is set to never, the Logical Domains Manager exits recovery mode without taking any action and the physical domain runs the factory-default configuration.
Note - If the system firmware requests recovery mode while it is not enabled, issue the following commands to enable recovery mode after the request is made:
primary# svccfg -s ldmd setprop ldmd/recovery_mode = astring: auto primary# svcadm refresh ldmd primary# svcadm restart ldmd
Recovery mode is initiated immediately in this scenario only if no changes were made to the system, that is, if it is still in the factory-default configuration.
In addition to enabling recovery mode, you can specify a timeout value for a root domain boot during recovery. By default, the ldmd/recovery_mode_boot_timeout property value is 30 minutes. Valid values start at 5 minutes.