|C H A P T E R 7|
Automatic Diagnosis and Recovery
This chapter describes the error diagnosis and domain recovery capabilities included with the firmware for Sun Fire midrange systems. This chapter explains the following:
The diagnosis and recovery features are enabled by default on Sun Fire midrange systems. This section provides an overview of how these features work.
Depending on the type of hardware errors that occur and the diagnostic controls that are set, the system controller performs certain diagnosis and domain recovery steps, as FIGURE 7-1 shows. The firmware includes an auto-diagnosis (AD) engine, which detects and diagnoses hardware errors that affect the availability of a platform and its domains.
The following summary describes the process shown in FIGURE 7-1:
1. System Controller detects domain hardware error and pauses the domain.
2. Auto-diagnosis. The AD engine analyzes the hardware error and determines which field-replaceable units (FRUs) are associated with the hardware error.
The AD engine provides one of the following diagnosis results, depending on the hardware error and the components involved:
The AD engine records the diagnosis information for the affected components and maintains this information as part of the component health status (CHS).
The AD reports diagnosis information through the following:
CODE EXAMPLE 7-1 shows an auto-diagnosis event message that appears on the platform console. In this example, a single FRU is responsible for the hardware error. See Reviewing Auto-Diagnosis Event Messages for details on the AD message contents.
The output from these commands supplements the diagnosis information presented in the platform and domain event messages and can be used for additional troubleshooting purposes.
3. Auto-restoration. During the auto-restoration process, POST reviews the component health status of FRUs that were updated by the AD engine. POST uses this information and tries to isolate the fault by deconfiguring (disabling) any FRUs from the domain that have been determined to cause the hardware error. Even if POST cannot isolate the fault, the system controller then automatically reboots the domain as part of domain restoration.
The system controller automatically monitors domains for hangs when either of the following occurs:
The default timeout value is three minutes, but you can override this value by setting the watchdog_timeout_seconds parameter in the domain /etc/systems file. If you set the value to less than three minutes, the system controller uses three minutes (the default value) as the timeout period. For details on this system parameter, refer to the system(4) man page of your Solaris operating environment release.
When the hang policy parameter of the setupdomain command is set to reset, the system controller automatically performs an externally initiated reset (XIR) and reboots the hung domain. If the OBP.error-reset-recovery parameter of the setupdomain command is set to sync, a core file is also generated after an XIR and can be used to troubleshoot the domain hang. See Domain Parameters section for details.
CODE EXAMPLE 7-2 shows the domain console message displayed when the domain heartbeat stops.
CODE EXAMPLE 7-3 shows the domain console message displayed when the domain does not respond to interrupts.
Starting with the 5.15.3 release, certain non-fatal domain hardware errors are identified by the Solaris operating environment and reported to the system controller. The system controller does the following:
The next time that POST is run, POST reviews the health status of affected resources and if possible, deconfigures the appropriate resources from the system.
CODE EXAMPLE 7-4 shows a domain diagnosis event message for a non-fatal domain error. When you see such event messages, contact your service provider so that the appropriate service action can be initiated. The event message information provided is described in Reviewing Auto-Diagnosis Event Messages.
You can obtain further information about components deconfigured by POST by using the showboards and showcomponent commands, as described in Reviewing Component Status.
This section explains the various controls and domain parameters that affect the domain restoration features.
Sun strongly recommends that you define platform and domain loghosts to which all system log (syslog) messages are forwarded and stored. Platform and domain messages, including the auto-diagnosis and domain-recovery event messages, cannot be stored locally. By specifying a loghost for platform and domain log messages, you can use the loghost to monitor and review critical events and messages as needed. However, you must set up a loghost server if you want to assign platform and domain loghosts.
You assign the loghosts through the Loghost and the Log Facility parameters in the setupplatform and setupdomain commands. The facility level identifies where the log messages originate, either the platform or a domain. For details on these commands, refer to their command descriptions in the Sun Fire Midrange System Controller Command Reference Manual.
TABLE 7-1 describes the domain parameter settings in the setupdomain command that control the diagnostic and domain recovery process. The default values for the diagnostic and domain recovery parameters are the recommended settings.
Note - If you do not use the default settings, the domain restoration features will not function as described in Automatic Diagnosis and Recovery Overview.
Enables or disables the system controller to send data about the current state of each CPU to the console, loghost facility, or to a URL specified by the reset-data-ftp-url option before resetting the system during a system hang. This allows system state data to be preserved if console data is being logged. The output format is the same as the format used by the showresetstate command when dumping the CPU state data for a hung system manually.
Controls the amount of information that the system controller sends to the console during a reset when log-reset-data is enabled. When enabled, this option produces the same result as using the showresetstate -v command.
Defines the maximum level of POST that runs automatically during repeated domain panics. POST level is escalated upon repeated panics until it runs the level specified in max-panic-diag-limit. If the domain panics again, it is placed in standby..
Automatically reboots the domain after an XIR occurs and generates a core file that can be used to troubleshoot the domain hang. However, be aware that sufficient disk space must be allocated in the domain swap area to hold the core file.
For a complete description of all the domain parameters and their values, refer to the setupdomain command description in the Sun Fire Midrange System Controller Command Reference Manual.
This section describes various ways to monitor hardware errors and obtain additional information about components associated with hardware errors.
Auto-diagnosis[AD] and domain [DOM] event messages are displayed on the platform and domain console and also in the following:
Each line of the loghost output contains a timestamp, a syslog ID number, and the facility level that identifies the platform or a domain where the log message originated.
In systems with SC V2s, certain log messages are maintained in persistent storage. You can selectively view certain types of persistent log messages according to message type, such as fault event messages, by using the showlogs -p -f filter command. For details, refer to the showlogs command description in the Sun Fire Midrange System Controller Command Reference Manual.
The diagnostic information logged on the platform and domain are similar, but the domain log provides additional information on the domain hardware error. The [AD] or [DOM] event messages (see CODE EXAMPLE 7-1, CODE EXAMPLE 7-4, CODE EXAMPLE 7-5, and CODE EXAMPLE 7-6) include the following information:
In some cases, be aware that not all the FRUs listed are necessarily faulty. The fault may reside in a subset of the components identified.
You can obtain additional information about components that have been deconfigured as part of the auto-diagnosis process or disabled for other reasons by reviewing the following items:
CODE EXAMPLE 7-7 shows the location assignments and the status for all components in the system. The diagnostic-related information is provided in the Status column for a component. Components that have a Failed or Disabled status are deconfigured from the system. The Failed status indicates that the board failed testing and is not usable. Disabled indicates that the board has been deconfigured from the system, because it was disabled using the setls command or because it failed POST. Degraded status indicates that certain components on the boards have failed or are disabled, but there are still usable parts on the board. Components with degraded status are configured into the system.
You can obtain additional information about Failed, Disabled, or Degraded components by reviewing the output from the showcomponent command.
The Status column in CODE EXAMPLE 7-8 shows the status for components. The status is either enabled or disabled. The disabled components are deconfigured from the system. The POST status chs (abbreviation for component health status) flags the component for further analysis by your service provider.
Note - Disabled components that have a POST status of chs cannot be enabled by using the setls command. Contact your service provider for assistance. In some cases, subcomponents belonging to a "parent" component associated with a hardware error will also reflect a disabled status, as will the parent. You cannot re-enable the subcomponents of a parent component associated with a hardware error. Review the auto-diagnosis event messages to determine which parent component is associated with the error.
For systems configured with SC V2s, the showerrorbuffer -p command displays the system error contents maintained in persistent storage.
However, for systems that do not have SC V2s, the showerrorbuffer command displays the contents of the system error buffer and displays error messages that otherwise might be lost when your domains are rebooted as part of the domain recovery process.
In either case, the information displayed can be used by your service provider for troubleshooting purposes.
CODE EXAMPLE 7-9 shows the output displayed for a domain hardware error, maintained in the system error buffer.