Sun Firetrademark 6800/4810/4800/3800 Systems Platform Administration Manual Table of ContentsPrevious ChapterNext ChapterBook Index


C H A P T E R  7
Diagnosis and Domain Restoration

This chapter describes the error diagnosis and domain restoration capabilities included with the firmware for Sun Fire 6800/4810/4800/3800 systems. This chapter explains the following:

Diagnosis and Domain Restoration Overview

The diagnosis and domain restoration features are enabled by default on Sun Fire midframe systems, starting with firmware version 5.15.0. This section provides an overview of how these capabilities work.

Auto-Diagnosis and Auto-Restoration

Depending on the type of hardware errors that occur and the diagnostic controls that are set, the system controller performs certain diagnosis and domain restoration steps, as FIGURE 7-1 shows. The firmware includes an auto-diagnosis (AD) engine, which detects and diagnoses hardware errors that affect the availability of a platform and its domains.

 FIGURE 7-1 Error Diagnosis and Domain Restoration Process

Diagram that shows the main steps in the error diagnosis and domain restoration process: Domain hardware error detection and domain pause, automatic diagnosis, and automatic domain restoration.

The following summary describes the process shown in FIGURE 7-1:

1. System Controller detects domain hardware error and pauses the domain.

2. Auto-diagnosis. The AD engine analyzes the hardware error and determines which field-replaceable units (FRUs) are associated with the hardware error.

The AD engine provides one of the following diagnosis results, depending on the hardware error and the components involved:
The AD engine records the diagnosis information for the affected components and maintains this information as part of the component health status (CHS).
The AD reports diagnosis information through the following:
CODE EXAMPLE 7-1 shows an auto-diagnosis event message that appears on the platform console. In this example, a single FRU is responsible for the hardware error. See Reviewing Auto-Diagnosis Event Messages for details on the AD message contents.
CODE EXAMPLE 7-1 Example of Auto-Diagnosis Event Message Displayed on the Platform Console 


Jan 23 20:47:11 schostname Platform.SC: ErrorMonitor: Domain A has a SYSTEM ERROR

.

.

.

[AD] Event: SF3800.ASIC.SDC.PAR_SGL_ERR.60111010

     CSN: 124H58EE  DomainID: A ADInfo: 1.SCAPP.15.0

     Time: Thu Jan 23 20:47:11 PST 2003

     FRU-List-Count: 1; FRU-PN: 5014362; FRU-SN: 011600; FRU-LOC: /N0/SB0

     Recommended-Action: Service action required

 

Jan 23 20:47:16 schostname Platform.SC: A fatal condition is detected on Domain 

A. Initiating automatic restoration for this domain.

 


Note - Contact your service provider when you see these auto-diagnosis messages. Your service provider will review the auto-diagnosis information and initiate the appropriate service action.


The output from these commands supplements the diagnosis information presented in the platform and domain event messages and can be used for additional troubleshooting purposes.

3. Auto-restoration. During the auto-restoration process, POST reviews the component health status of FRUs that were updated by the AD engine. POST uses this information and tries to isolate the fault by deconfiguring (disabling) any FRUs from the domain that have been determined to cause the hardware error. Even if POST cannot isolate the fault, the system controller then automatically reboots the domain as part of domain restoration.

Automatic Recovery of Hung Domains

The system controller automatically monitors domains for hangs when

  • A domain heartbeat stops within a designated timeout period.

The default timeout value is three minutes, but you can override this value by setting the watchdog_timeout_seconds parameter in the domain /etc/systems file. If you set the value to less than three minutes, the system controller uses three minutes (the default value) as the timeout period. For details on this system parameter, refer to the system(4) man page of your Solaris operating environment release.
  • The domain does not respond to interrupts.

When the hang policy parameter of the setupdomain command is set to reset, the system controller automatically performs an externally initiated reset (XIR) and reboots the hung domain. If the OBP.error-reset-recovery parameter of the setupdomain command is set to sync, a core file is also generated after an XIR and can be used to troubleshoot the domain hang. See Domain Parameters section for details.

CODE EXAMPLE 7-2 shows the domain console message displayed when the domain heartbeat stops. 

Jan 22 14:59:23 schostname Domain-A.SC: Domain watchdog timer expired.

Jan 22 14:59:23 schostname Domain-A.SC: Using default hang-policy (RESET).

Jan 22 14:59:23 schostname Domain-A.SC: Resetting (XIR) domain.

 
CODE EXAMPLE 7-2 Example of Domain Message Output for Automatic Domain Recovery After the Domain Heartbeat Stops

CODE EXAMPLE 7-3 shows the domain console message displayed when the domain does not respond to interrupts. 

Jan 22 14:59:23 schostname Domain-A.SC: Domain is not responding to interrupts.

Jan 22 14:59:23 schostname Domain-A.SC: Using default hang-policy (RESET).

Jan 22 14:59:23 schostname Domain-A.SC: Resetting (XIR) domain.

 
CODE EXAMPLE 7-3 Example of Domain Console Output for Automatic Domain Recovery After the Domain Does Not Respond to Interrupts

Domain Restoration Controls

This section explains the various controls and domain parameters that affect the domain restoration features.

The syslog Loghost

Sun strongly recommends that you define platform and domain loghosts to which all system log (syslog) messages are forwarded and stored. Platform and domain messages, including the auto-diagnostic and domain-restoration event messages, cannot be stored locally. By specifying a loghost for platform and domain log messages, you can use the loghost to monitor and review critical events and messages as needed. However, you must set up a loghost server if you want to assign platform and domain loghosts.

You assign the loghosts through the Loghost and the Log Facility parameters in the setupplatform and setupdomain commands. The facility level identifies where the log messages originate, either the platform or a domain. For details on these commands, refer to their command descriptions in the Sun Fire 6800/4810/4800/3800 System Controller Command Reference Manual.

Domain Parameters

TABLE 7-1 describes the domain parameter settings in the setupdomain command that control the diagnostic and domain recovery process. The default values for the diagnostic and domain restoration parameters are the recommended settings.


Note - If you do not use the default settings, the domain restoration features will not function as described in Diagnosis and Domain Restoration Overview.


TABLE 7-1 Diagnostic and Domain Recovery Parameters in the setupdomain Command

setupdomain Parameter

Default Value

Description

reboot-on-error

true

Automatically reboots the domain when a hardware error is detected. Also boots the Solaris operating environment when the OBP.auto-boot parameter is set to true.

hang-policy

reset

Automatically resets a hung domain through an externally initiated reset (XIR).

OBP.auto-boot

true

Boots the Solaris operating environment after POST runs.

OBP.error-reset-recovery

sync

Automatically reboots the domain after an XIR occurs and generates a core file that can be used to troubleshoot the domain hang. However, be aware that sufficient disk space must be allocated in the domain swap area to hold the core file.


For a complete description of all the domain parameters and their values, refer to the setupdomain command description in the Sun Fire 6800/4810/4800/3800 System Controller Command Reference Manual.

Obtaining Auto-Diagnosis and Domain Restoration Information

This section describes various ways to monitor diagnostic errors and obtain additional information about components that are associated with hardware errors.

Reviewing Auto-Diagnosis Event Messages

Auto-diagnosis event messages are displayed on the platform and domain console and also in the following:

  • The platform or domain loghost, provided that you have defined the syslog host for the platform and domains.

Each line of the loghost output contains a timestamp, a syslog ID number, and the facility level that identifies where the log message originated (the platform or a domain).
  • The showlogs command output, which displays the event messages logged on the platform or domain console.

For details on the showlogs command, refer to the command description in the Sun Fire 6800/4810/4800/3800 System Controller Command Reference Manual.

The diagnostic information logged on the platform and domain are similar, but the domain log provides additional information on the domain hardware error. The auto-diagnosis event message includes the following information:

  • [AD]- Beginning of the auto-diagnosis message.

  • Event - An alphanumeric text string that identifies the platform and event-specific information used by your service provider.

  • CSN - Chassis serial number.

  • DomainID - The domain affected by the hardware error.

  • ADInfo - The version of the auto-diagnosis message, the name of the diagnosis engine (SCAPP), and the auto-diagnosis engine version.

  • Time - The day of the week, month, date, time (hours, minutes, and seconds), time zone, and year of the auto-diagnosis.

  • FRU-List-Count - The number of components (FRUs) involved with the error and the following FRU data:

    • If a single component is implicated, the FRU part number, serial number, and location are displayed, as CODE EXAMPLE 7-1 shows.

    • If multiple components are implicated, the FRU part number, serial number, and location for all the components involved are reported, as CODE EXAMPLE 7-4 shows.

In some cases, be aware that not all the FRUs listed are necessarily faulty. The fault may reside in a subset of the components identified.
    • If the AD engine cannot implicate specific components, the term UNRESOLVED is displayed, as CODE EXAMPLE 7-5 shows.

  • Recommended-Action: Service action required - Instructs the platform or domain administrator to contact their service provider for further service action. Also indicates the end of the auto-diagnosis message.

  

Jan. 23 21:07:51 schostname Domain-A.SC: ErrorMonitor: Domain A has a SYSTEM ERROR

.

.

.

[AD] Event: SF3800.ASIC.SDC.PAR_L2_ERR_TT.60113022

     CSN: 124H58EE  DomainID: A ADInfo: 1.SCAPP.15.0

     Time: Thu Jan 23 21:07:51 PST 2003

     FRU-List-Count: 2; FRU-PN: 5015876; FRU-SN: 000429; FRU-LOC: RP0        

                        FRU-PN: 5014362; FRU-SN: 011570; FRU-LOC: /N0/SB2

     Recommended-Action: Service action required

Jan 23 21:08:01 schostname Domain-A.SC: A fatal condition is detected on Domain 

A. Initiating automatic restoration for this domain.

 
CODE EXAMPLE 7-4 Example of Domain Console Auto-Diagnostic Message Involving Multiple FRUs

  

Jan 23 21:47:28 schostname Domain-A.SC: ErrorMonitor: Domain A has a SYSTEM ERROR

.

.

.

[AD] Event: SF3800

     CSN: 124H58EE  DomainID: A ADInfo: 1.SCAPP.15.0

     Time: Thu Jan 23 21:47:28 PST 2003

     FRU-List-Count: 0; FRU-PN: ; FRU-SN: ; FRU-LOC: UNRESOLVED

     Recommended-Action: Service action required

 

Jan 23 21:47:28 schostname Domain-A.SC: A fatal condition is detected on Domain

A. Initiating automatic restoration for this domain.

 
CODE EXAMPLE 7-5 Example of Domain Console Auto-Diagnostic Message Involving an Unresolved Diagnosis

Reviewing Component Status

You can obtain additional information about components that have been deconfigured as part of the auto-diagnosis process or disabled for other reasons by reviewing the following items:

  • The showboards command output after an auto-diagnosis has occurred

CODE EXAMPLE 7-6 shows the location assignments and the status for all components in the system. The diagnostic-related information is provided in the Status column for a component. Components that have a Failed or Disabled status are deconfigured from the system. The Failed status indicates that the board failed testing and is not usable. Disabled indicates that the board has been deconfigured from the system, because it was disabled using the setls command or because it failed POST. Degraded status indicates that certain components on the boards have failed or are disabled, but there are still usable parts on the board. Components with degraded status are configured into the system.
You can obtain additional information about Failed, Disabled, or Degraded components by reviewing the output from the showcomponent command.
schostname: SC> showboards

 

Slot     Pwr Component Type                 State      Status     Domain

----     --- --------------                 -----      ------     ------

SSC0     On  System Controller              Main       Passed     -

SSC1     -   Empty Slot                     -          -          -

ID0      On  Sun Fire 4800 Centerplane      -          OK         -

PS0      -   Empty Slot                     -          -          -

PS1      On  A153 Power Supply              -          OK         -

PS2      On  A153 Power Supply              -          OK         -

FT0      On  Fan Tray                       High Speed OK         -

FT1      On  Fan Tray                       High Speed OK         -

FT2      On  Fan Tray                       High Speed OK         -

RP0      On  Repeater Board                 -          OK         -

/N0/SB0  On  CPU Board                      Assigned   Disabled   A

SB2      -   Empty Slot                     Assigned   -          A

/N0/SB4  On  CPU Board                      Active     Degraded   A

/N0/IB6  On  PCI I/O Board                  Active     Passed     A

IB8      Off PCI I/O Board                  Available  Not tested Isolated

 
CODE EXAMPLE 7-6 showboards Command Output - Disabled and Degraded Components

  • The showcomponent command output after an auto-diagnosis has occurred

The Status column in CODE EXAMPLE 7-7 shows the status for components. The status is either enabled or disabled. The disabled components are deconfigured from the system. The POST status chs (abbreviation for component health status) flags the component for further analysis by your service provider.

Note - Disabled components that have a POST status of chs cannot be enabled by using the setls command. Contact your service provider for assistance. In some cases, subcomponents belonging to a "parent" component associated with a hardware error will also reflect a disabled status, as does the parent. You cannot re-enable the subcomponents of a parent component associated with a hardware error. Review the auto-diagnosis event messages to determine which parent component is associated with the error. 


schostname: SC> showcomponent

 

Component          Status    Pending POST   Description

---------          ------    ------- ----   -----------

/N0/SB0/P0         disabled  -       chs    UltraSPARC-III+, 900MHz, 8M ECache

/N0/SB0/P1         disabled  -       chs    UltraSPARC-III+, 900MHz, 8M ECache

/N0/SB0/P2         disabled  -       chs    UltraSPARC-III+, 900MHz, 8M ECache

/N0/SB0/P3         disabled  -       chs    UltraSPARC-III+, 900MHz, 8M ECache

/N0/SB0/P0/B0/L0   disabled  -       chs    empty

/N0/SB0/P0/B0/L2   disabled  -       chs    empty

/N0/SB0/P0/B1/L1   disabled  -       chs    512M DRAM

/N0/SB0/P0/B1/L3   disabled  -       chs    512M DRAM

.

.

.

/N0/SB0/P3/B0/L0    disabled -       chs    empty

/N0/SB0/P3/B0/L2    disabled -       chs    empty

/N0/SB0/P3/B1/L1    disabled -       chs    512M DRAM

/N0/SB0/P3/B1/L3    disabled -       chs    512M DRAM

/N0/SB4/P0          enabled  -       pass   UltraSPARC-III+, 900MHz, 8M ECache

/N0/SB4/P1          enabled  -       pass   UltraSPARC-III+, 900MHz, 8M ECache

/N0/SB4/P2          enabled  -       pass   UltraSPARC-III+, 900MHz, 8M ECache

/N0/SB4/P3          enabled  -       pass   UltraSPARC-III+, 900MHz, 8M ECache

.

.

.

 
CODE EXAMPLE 7-7 showcomponent Command Output - Disabled Components

Reviewing Additional Error Information

The showerrorbuffer command shows the contents of the system error buffer and displays error messages that otherwise might be lost when your domains are rebooted as part of the domain recovery process. The information displayed can be used by your service provider for troubleshooting purposes.

CODE EXAMPLE 7-8 shows the output displayed for a domain hardware error. 

schostname: SC> showerrorbuffer

 

ErrorData[0]

  Date: Tue Jan 21 14:30:20 PST 2003

  Device: /SSC0/sbbc0/systemepld

  Register: FirstError[0x10] : 0x0200

            SB0 encountered the first error

ErrorData[1]

  Date: Tue Jan 21 14:30:20 PST 2003

  Device: /partition0/domain0/SB4/bbcGroup0/repeaterepld

  Register: FirstError[0x10]: 0x00c0

            sbbc0 encountered the first error

            sbbc1 encountered the first error

ErrorData[2]

  Date: Tue Jan 21 14:30:20 PST 2003

  Device: /partition0/domain0/SB4/bbcGroup0/sbbc0

  ErrorID: 0x50121fff

  Register: ErrorStatus[0x80] : 0x00000300

	          SafErr [09:08] : 0x3 Fireplane device asserted an error

.

.

.

 
CODE EXAMPLE 7-8 showerrorbuffer Command Output - Hardware Error



  Sun Firetrademark 6800/4810/4800/3800 Systems Platform Administration Manual 817-0999-10 Table of ContentsPrevious ChapterNext ChapterBook Index


Copyright © 2003, Sun Microsystems, Inc. All rights reserved.