|C H A P T E R 7|
Automatic Diagnosis and Recovery
This chapter describes the error diagnosis and domain recovery capabilities included with the firmware for Sun Fire entry-level midrange systems.
This chapter explains the following topics:
The diagnosis and recovery features are enabled by default on Sun Fire midrange systems. This section provides an overview of how these features work.
Depending on the type of hardware errors that occur and the diagnostic controls that are set, the system controller performs certain diagnosis and recovery steps, as FIGURE 7-1 shows. The firmware includes an auto-diagnosis (AD) engine, which detects and diagnoses hardware errors that affect the availability of a system.
|Note - Although entry-level midrange systems do not support the multiple domains that other midrange systems support, by convention, diagnostic output provides system status as the status for Domain A.|
FIGURE 7-1 Auto Diagnosis and Recovery Process
[ D ]
The following summary describes the process shown in FIGURE 7-1:
1. The SC detects hardware error and pauses the operating system.
2. Auto-diagnosis. The AD engine analyzes the hardware error and determines which field-replaceable units (FRUs) are associated with the hardware error.
The AD engine provides one of the following diagnosis results, depending on the hardware error and the components involved:
The AD engine records the diagnosis information for the affected components and maintains this information as part of the component health status (CHS).
The AD reports diagnosis information through console event messages.
|Note - Contact your service provider when you see these auto-diagnosis messages. Your service provider will review the auto-diagnosis information and initiate the appropriate service action.|
The output from these commands supplements the diagnosis information presented in the event messages and can be used for additional troubleshooting purposes.
3. Auto-restoration. During the auto-restoration process, POST reviews the component health status of FRUs that were updated by the AD engine. POST uses this information and tries to isolate the fault by deconfiguring (disabling) any FRUs from the domain that have been determined to cause the hardware error. Even if POST cannot isolate the fault, the system controller then automatically reboots the domain as part of domain restoration.
|Note - To take advantage of the automatic recovery feature, make sure that the Openboot PROM variable hang-policy is set to reset.|
The system controller automatically monitors systems for hangs when either of the following occurs:
The default timeout value is three minutes, but you can override this value by setting the watchdog_timeout_seconds parameter in the domain /etc/systems file. If you set the value to less than three minutes, the system controller uses three minutes (the default value) as the timeout period. For details on this system parameter, see the system(4) man page of your Solaris Operating System release.
When the host watchdog (as described in the setupsc command) is enabled, the system controller automatically performs an externally initiated reset (XIR) and reboots the hung operating system. If the OpenBoot PROM nvram variable, error-reset-recovery is set to sync, a core file is also generated after an XIR and can be used to troubleshoot the operating system hang.
CODE EXAMPLE 7-2 shows the console message displayed when the operating system heartbeat stops.
CODE EXAMPLE 7-3 shows the console message displayed when the operating system does not respond to interrupts.
Certain nonfatal hardware errors are identified by the Solaris Operating System and reported to the system controller. The system controller does the following:
The next time that POST is run, POST reviews the health status of affected resources and if possible, deconfigures the appropriate resources from the system.
CODE EXAMPLE 7-4 shows an event message for a nonfatal domain error. When you see such event messages, contact your service provider so that the appropriate service action can be initiated. The event message information provided is described in Reviewing Auto-Diagnosis Event Messages.
You can obtain further information about components deconfigured by POST by using the showboards and showcomponent commands, as described in Reviewing Component Status.
This section explains the various controls and parameters that affect the restoration features.
TABLE 7-1 describes the parameter settings that control the diagnostic and operating system recovery process. The default values for the diagnostic and operating system recovery parameters are the recommended settings.
|Note - If you do not use the default settings, the restoration features will not function as described in Automatic Diagnosis and Recovery Overview.|
The Solaris 10 Operating System incorporates features that automatically isolate faulty parts of such memory modules, thus avoiding the need to completely disable these modules and increasing system availability.
Automatically reboots the system after an XIR occurs and generates a core file that can be used to troubleshoot the system hang. However, be aware that sufficient disk space must be allocated in the swap area to hold the core file.
This section describes various ways to monitor hardware errors and obtain additional information about components associated with hardware errors.
Auto-diagnosis [AD] and domain [DOM] event messages are displayed on the console and also in the following:
In systems with enhanced-memory system controllers (SC V2s), log messages are maintained in a persistent buffer. You can selectively view certain types of log messages according to message type, such as fault event messages, by using the showlogs -p -f filter command. For details, see the showlogs command description in the Sun Fire Entry-Level Midrange System Controller Command Reference Manual.
The [AD] or [DOM] event messages (see CODE EXAMPLE 7-1, CODE EXAMPLE 7-4, CODE EXAMPLE 7-5, and CODE EXAMPLE 7-6) include the following information:
In some cases, be aware that not all the FRUs listed are necessarily faulty. The fault may reside in a subset of the components identified.
Tue Dec 02 14:35:56 commando lom: ErrorMonitor: Domain A has a SYSTEM ERROR . . . Tue Dec 02 14:35:59 commando lom: [AD] Event: E2900 CSN: DomainID: A ADInfo: 1.SCAPP.17.0 Time: Tue Dec 02 14:35:57 PST 2003 FRU-List-Count: 0; FRU-PN: ; FRU-SN: ; FRU-LOC: UNRESOLVED Recommended-Action: Service action required Tue Dec 02 14:35:59 commando lom: A fatal condition is detected on Domain A. Initiating automatic restoration for this domain
You can obtain additional information about components that have been unconfigured as part of the auto-diagnosis process or disabled for other reasons by reviewing the following items:
CODE EXAMPLE 7-6 shows the location assignments and the status for all components in the system. The diagnostic-related information is provided in the Status column for a component. Components that have a Failed or Disabled status are deconfigured from the system. The Failed status indicates that the board failed testing and is not usable. Disabled indicates that the board has been deconfigured from the system, because it was disabled using the setls command or because it failed POST. Degraded status indicates that certain components on the boards have failed or are disabled, but there are still usable parts on the board. Components with degraded status are configured into the system.
You can obtain additional information about Failed, Disabled, or Degraded components by reviewing the output from the showcomponent command.
Slot Pwr Component Type State Status ---- --- -------------- ----- ------ SSC1 On System Controller V2 Main Passed /N0/SCC - System Config Card Assigned OK /N0/BP - Baseplane Assigned Passed /N0/SIB - Indicator Board Assigned Passed /N0/SPDB - System Power Distribution Bd. Assigned Passed /N0/PS0 On A166 Power Supply - OK /N0/PS1 On A166 Power Supply - OK /N0/PS2 On A166 Power Supply - OK /N0/PS3 On A166 Power Supply - OK /N0/FT0 On Fan Tray Auto Speed Passed /N0/RP0 On Repeater Board Assigned OK /N0/RP2 On Repeater Board Assigned OK /N0/SB0 On CPU Board Active Passed /N0/SB2 On CPU Board V3 Assigned Disabled /N0/SB4 On CPU Board Active Degraded /N0/IB6 On PCI I/O Board Active Passed /N0/MB - Media Bay Assigned Passed
The Status column in CODE EXAMPLE 7-7 shows the status for components. The status is either enabled or disabled. The disabled components are deconfigured from the system. The POST status chs (abbreviation for component health status) flags the component for further analysis by your service provider.
|Note - Disabled components that have a POST status of chs cannot be enabled by using the setls command. Contact your service provider for assistance. In some cases, subcomponents belonging to a “parent” component associated with a hardware error will also reflect a disabled status, as will the parent. You cannot re-enable the subcomponents of a parent component associated with a hardware error. Review the auto-diagnosis event messages to determine which parent component is associated with the error.|
schostname: SC> showcomponent Component Status Pending POST Description --------- ------ ------- ---- ----------- /N0/SB0/P0 disabled - chs UltraSPARC-IV, 1050MHz, 16M ECache /N0/SB0/P1 disabled - chs UltraSPARC-IV, 1050MHz, 16M ECache /N0/SB0/P2 disabled - chs UltraSPARC-IV, 1050MHz, 16M ECache /N0/SB0/P3 disabled - chs UltraSPARC-IV, 1050MHz, 16M ECache /N0/SB0/P0/B0/L0 disabled - chs empty /N0/SB0/P0/B0/L2 disabled - chs empty /N0/SB0/P0/B1/L1 disabled - chs 2048M DRAM /N0/SB0/P0/B1/L3 disabled - chs 2048M DRAM . . . /N0/SB0/P3/B0/L0 disabled - chs empty /N0/SB0/P3/B0/L2 disabled - chs empty /N0/SB0/P3/B1/L1 disabled - chs 1024M DRAM /N0/SB0/P3/B1/L3 disabled - chs 1024M DRAM /N0/SB4/P0 enabled - pass UltraSPARC-IV, 1050MHz, 16M ECache /N0/SB4/P1 enabled - pass UltraSPARC-IV, 1050MHz, 16M ECache /N0/SB4/P2 enabled - pass UltraSPARC-IV, 1050MHz, 16M ECache /N0/SB4/P3 enabled - pass UltraSPARC-IV, 1050MHz, 16M ECache . . .
For systems configured with enhanced-memory SCs (SC V2s), the showerrorbuffer -p command shows the system error contents maintained in the persistent buffer.
However, for systems that do not have enhanced-memory SCs, the showerrorbuffer command shows the contents of the dynamic buffer and displays error messages that otherwise might be lost when your domains are rebooted as part of the domain recovery process.
In either case, the information displayed can be used by your service provider for troubleshooting purposes.
CODE EXAMPLE 7-8 shows the output displayed for a domain hardware error.
EX07: lom>showerrorbuffer ErrorData Date: Fri Jan 30 10:23:32 EST 2004 Device: /SSC1/sbbc0/systemepld Register: FirstError[0x10] : 0x0200 SB0 encountered the first error ErrorData Date: Fri Jan 30 10:23:32 EST 2004 Device: /SB0/bbcGroup0/repeaterepld Register: FirstError[0x10]: 0x0002 sdc0 encountered the first error ErrorData Date: Fri Jan 30 10:23:32 EST 2004 Device: /SB0/sdc0 ErrorID: 0x60171010 Register: SafariPortError0[0x200] : 0x00000002 ParSglErr [01:01] : 0x1 ParitySingle error