C H A P T E R 2 - Managing RAS Features and System Firmware

C H A P T E R 2

Managing RAS Features and System Firmware

This chapter describes how to manage reliability, availability, and serviceability (RAS) features and system firmware, including Sun Advanced Lights Out Manager (ALOM) system controller, and automatic system recovery (ASR). In addition, this chapter describes how to unconfigure and reconfigure a device manually, and introduces multipathing software.

This chapter contains the following sections:

ALOM CMT and The System Controller

Automatic System Recovery

Unconfiguring and Reconfiguring Devices

Multipathing Software

Note - This chapter does not cover detailed troubleshooting and diagnostic procedures. For information about fault isolation and diagnostic procedures, refer to the diagnostics and troubleshooting guide for your server.

ALOM CMT and The System Controller

The system controller supports a total of nine concurrent ALOM CMT sessions per server, one connection through the serial management port and eight connections available through the network management port.

After you log in to your ALOM account, the system controller command prompt (sc>) appears, and you can enter system controller commands. If the command you want to use has multiple options, you can either enter the options individually or grouped together, as shown in the following example. The commands shown in the following example are identical.

sc> poweroff -f -y

sc> poweroff -fy

Logging In To The System Controller

All environmental monitoring and control is handled by the system controller. The system controller command prompt (sc>) provides you with a way of interacting with the system controller. For more information about the sc> prompt, see ALOM CMT and The sc> Prompt.

For instructions on connecting to the system controller, see:

Accessing the System Controller

Activating the Network Management Port

Note - This procedure assumes that the system console is directed to use the serial management and network management ports (the default configuration).

To Log In To ALOM CMT

1. If you are logged in to the system console, type #. (Hash-Period) to get to the sc> prompt.

Press the Hash key, followed by the Period key. Then press the Return key.

2. At the ALOM CMT login prompt, enter the login name and press Return.

The default login name is admin.

Sun(tm) Advanced Lights Out Manager CMT v1.3

Please login: admin

3. At the password prompt, enter the password and press Return to get to the sc> prompt.

Please Enter password:

sc>

Note - There is no default password when connecting to ALOM CMT for the first time using the serial management port. When connecting to the system controller using the network management port for the first time, the default ALOM CMT password is the last 8 digits of the Chassis Serial Number. The Chassis Serial Number can be found printed on the back of the server or in the printed system information sheet which shipped with your server. You must assign a password during initial system configuration. For more information, refer to the installation guide for your server and the ALOM CMT guide for your server.

Caution - To provide optimum system security, change the default system login name and password during initial setup.

Using the system controller, you can monitor the system, turn the Locator LED on and off, or perform maintenance tasks on the system controller card itself. For more information, refer to the ALOM CMT guide for your server.

To View Environmental Information

1. Log in to the system controller.

2. Use the showenvironment command to display a snapshot of the server's environmental status.

The information this command can display includes temperature, power supply status, front panel LED status, and so on.

Note - Some environmental information might not be available when the server is in Standby mode.

Note - You do not need system controller user permissions to use this command.

Interpreting System LEDs

The behavior of LEDs on your server conform the American National Standards Institute (ANSI) Status Indicator Standard (SIS). These standard LED behaviors are described in TABLE 2-1.

TABLE 2-1 LED Behavior and Meaning
LED Behavior	Meaning
Off	The condition represented by the color is not true.
Steady On	The condition represented by the color is true.
Standby Blink	The system is functioning at a minimal level and ready to resume full function.
Slow Blink	Transitory activity or new activity represented by the color is taking place.
Fast Blink	Attention is required.
Feedback Flash	Activity is taking place commensurate with the flash rate (such as disk drive activity).

The LEDs have assigned meanings, described in TABLE 2-2.

TABLE 2-2 LED Behaviors With Assigned Meanings
Color	Behavior	Definition	Description
White	Off	Steady State
	Fast Blink	4 Hz repeating sequence, equal intervals On and Off.	This indicator helps you to locate a particular enclosure, board, or subsystem (for example, the Locator LED).
Blue	Off	Steady State
	Steady On	Steady State	If blue is on, a service action can be performed on the applicable component with no adverse consequences (for example, the OK-to-Remove LED).
Yellow/Amber	Off	Steady State
	Steady On	Steady State	This indicator signals the existence of a fault condition. Service is required (for example, the Service Required LED).
Green	Off	Steady State
	Standby Blink	Repeating sequence consisting of a brief (0.1 sec.) on flash followed by a long off period (2.9 sec.)	The system is running at a minimum level and is ready to be quickly revived to full function (for example, the System Activity LED).
	Steady On	Steady State	Status normal; system or component functioning with no service actions required
	Slow Blink		A transitory (temporary) event is taking place for which direct proportional feedback is not needed or not feasible.

Controlling the Locator LED

You control the Locator LED from the sc> prompt or by the Locator button on the front of the chassis.

FIGURE 2-1 Locator Button on the Front of the Server Chassis

Graphic image of the front panel of the Sun Fire T1000 server. The locator button is located in the upper left corner of the chassis.

To turn on the Locator LED from the system controller command prompt, type:

sc> setlocator on

To turn off the Locator LED from the system controller command prompt, type:

sc> setlocator off

To display the state of the Locator LED from the system controller command prompt, type:

sc> showlocator

Locator LED is on.

Note - You do not need user permissions to use the setlocator and showlocator commands

Automatic System Recovery

The system provides for automatic system recovery (ASR) from failures in memory modules or PCI cards.

Automatic system recovery functionality enables the system to resume operation after experiencing certain nonfatal hardware faults or failures. When ASR is enabled, the system's firmware diagnostics automatically detect failed hardware components. An autoconfiguring capability designed into the system firmware enables the system to unconfigure failed components and to restore system operation. As long as the system is capable of operating without the failed component, the ASR features enable the system to reboot automatically, without operator intervention.

Note - ASR is not activated until you enable it. See Enabling and Disabling Automatic System Recovery.

For more information about ASR, refer to the service manual for your server.

AutoBoot Options

The system firmware stores a configuration variable called auto-boot?, which controls whether the firmware will automatically boot the operating system after each reset. The default setting for Sun platforms is true.

Normally, if a system fails power-on diagnostics, auto-boot? is ignored and the system does not boot unless an operator boots the system manually. An automatic boot is generally not acceptable for booting a system in a degraded state. Therefore, the server's OpenBoot firmware provides a second setting, auto-boot-on-error?. This setting controls whether the system will attempt a degraded boot when a subsystem failure is detected. Both the auto-boot? and auto-boot-on-error? switches must be set to true to enable an automatic degraded boot.

To enable an Automatic Degraded Boot

Set the switches by typing:

ok setenv auto-boot? true

ok setenv auto-boot-on-error? true

Note - The default setting for auto-boot-on-error? is false. The system will not attempt a degraded boot unless you change this setting to true. In addition, the system will not attempt a degraded boot in response to any fatal nonrecoverable error, even if degraded booting is enabled. For examples of fatal nonrecoverable errors, see Error Handling Summary.

Error Handling Summary

Error handling during the power-on sequence falls into one of the following three cases:

If no errors are detected by POST or OpenBoot Diagnostics, the system attempts to boot if auto-boot? is true.

If only nonfatal errors are detected by POST or OpenBoot Diagnostics, the system attempts to boot if auto-boot? is true and auto-boot-on-error? is true. Nonfatal errors include the following:

Ethernet interface failure.

Serial interface failure.

PCI-Express card failure.

Memory failure. When a DIMM fails, the firmware unconfigures the entire logical bank associated with the failed module. Another nonfailing logical bank must be present in the system for the system to attempt a degraded boot. Note that certain DIMM failures might not be diagnosable to a single DIMM. These failures are fatal, and result in both logical banks being unconfigured.

Note - If POST or OpenBoot Diagnostics detect a nonfatal error associated with the normal boot device, the OpenBoot firmware automatically unconfigures the failed device and tries the next-in-line boot device, as specified by the boot-device configuration variable.

If a fatal error is detected by POST or OpenBoot Diagnostics, the system does not boot regardless of the settings of auto-boot? or auto-boot-on-error?. Fatal nonrecoverable errors include the following:

Any CPU failed

All logical memory banks failed

Flash RAM cyclical redundancy check (CRC) failure

Critical field-replaceable unit (FRU) PROM configuration data failure

Critical system configuration SEEPROM read failure

Critical application-specific integrated circuit (ASIC) failure

For more information about troubleshooting fatal errors, refer to the service manual for your server.

Reset Scenarios

Three ALOM CMT configuration variables, diag_mode, diag_level, and diag_trigger, control whether the system runs firmware diagnostics in response to system reset events.

The standard system reset protocol bypasses POST completely unless the virtual keyswitch or ALOM CMT variables are set as follows:

TABLE 2-3 Virtual Keyswitch Setting for Reset Scenario
Keyswitch	Value
virtual keyswitch	`diag`

TABLE 2-4 ALOM CMT Variable Settings for Reset Scenario
Variable	Value	Default
`diag-mode`	`normal` or `service`	`normal`
`diag-level`	`min` or `max`	`max`
`diag-trigger`	`power-on-reset error-reset`	`power-on-reset`

Therefore, ASR is enabled by default. For instructions, see Enabling and Disabling Automatic System Recovery.

Automatic System Recovery User Commands

The ALOM CMT commands are available for obtaining ASR status information and for manually unconfiguring or reconfiguring system devices. For more information, see:

Unconfiguring and Reconfiguring Devices

To Reconfigure a Device Manually

Obtaining Automatic System Recovery Information

Enabling and Disabling Automatic System Recovery

The automatic system recovery (ASR) feature is not activated until you enable it. Enabling ASR requires changing configuration variables in ALOM CMT as well as OpenBoot firmware.

To Enable Automatic System Recovery

1. At the sc> prompt, type:

sc> setsc diag-mode normal

sc> setsc diag-level max

sc> setsc diag-trigger power-on-reset

2. At the ok prompt, type:

ok setenv auto-boot true

ok setenv auto-boot-on-error? true

Note - For more information about OpenBoot configuration variables, refer to the Sun SPARC Enterprise T1000 Server Service Manual.

3. To cause the parameter changes to take effect, type:

ok reset-all

The system permanently stores the parameter changes and boots automatically when the OpenBoot configuration variable auto-boot? is set to true (its default value).

Note - To store parameter changes, you can also power cycle the system using the front panel Power button.

To Disable Automatic System Recovery

1. At the ok prompt, type:

ok setenv auto-boot-on-error? false

2. To cause the parameter changes to take effect, type:

ok reset-all

The system permanently stores the parameter change.

Note - To store parameter changes, you can also power cycle the system using the front panel Power button.

After you disable the automatic system recovery (ASR) feature, it is not activated again until you re-enable it.

Obtaining Automatic System Recovery Information

Use the following procedure to retrieve information about the status of system components affected by automatic system recovery (ASR).

At the sc> prompt, type:

sc> showcomponent

In the showcomponent command output, any devices marked disabled have been manually unconfigured using the system firmware. The showcomponent command also lists devices that have failed firmware diagnostics and have been automatically unconfigured by the system firmware.

For more information, see:

Automatic System Recovery

Enabling and Disabling Automatic System Recovery

To Disable Automatic System Recovery

Unconfiguring and Reconfiguring Devices

To Reconfigure a Device Manually

Unconfiguring and Reconfiguring Devices

To support a degraded boot capability, the ALOM CMT firmware provides the disablecomponent command, which enables you to unconfigure system devices manually. This command flags the specified device as disabled by creating an entry in the ASR database.

To Unconfigure a Device Manually

At the sc> prompt, type:

sc> disablecomponent asr-key

The asr-key is one of the device identifiers from TABLE 2-5

Note - The device identifiers are not case-sensitive. You can type them as uppercase or lowercase characters.

TABLE 2-5 Device Identifiers and Devices
Device Identifiers	Devices
`MB/CMP`cpu-number`/P`strand_number	CPU strand (Number: 0-31)
`PCIE`slot-number	PCI-E slot (Number: 0)
`MB/PCIEa`	PCI-E leaf A (`/pci@780`)
`MB/PCIEb`	PCI-E leaf B (`/pci@7c0`)
`MB/CMP0/CH`channel-number/Rrank-number/Ddimm-number	DIMMS

To Reconfigure a Device Manually

1. At the sc> prompt, type:

sc> enablecomponent asr-key

where asr-key is any device identifier from TABLE 2-5

Note - The device identifiers are not case-sensitive. You can type them as uppercase or lowercase characters.

You can use the ALOM CMT enablecomponent command to reconfigure any device that you previously unconfigured with the disablecomponent command.

Displaying System Fault Information

ALOM CMT software enables you to display current valid system faults. The showfaults command displays the fault ID, the faulted FRU device, and the fault message to standard output. The showfaults command also displays POST results.

To Display System Fault Information

Type showfaults

For example:

sc> showfaults

   ID FRU        Fault

    0 FT0.F2     SYS_FAN at FT0.F2 has FAILED.

Adding the -v option displays additional information,

sc> showfaults -v

   ID Time              FRU        Fault

    0   MAY 20 10:47:32 FT0.F2     SYS_FAN at FT0.F2 has FAILED.

For more information about the showfaults command, refer to the Advanced Lights Out Management (ALOM) CMT v1.3 Guide.

Multipathing Software

Multipathing software enables you to define and control redundant physical paths to I/O devices, such as storage devices and network interfaces. If the active path to a device becomes unavailable, the software can automatically switch to an alternate path to maintain availability. This capability is known as automatic failover. To take advantage of multipathing capabilities, you must configure the server with redundant hardware, such as redundant network interfaces or two host bus adapters connected to the same dual-ported storage array.

For your server, three different types of multipathing software are available:

Solaris IP Network Multipathing software provides multipathing and load-balancing capabilities for IP network interfaces.

VERITAS Volume Manager (VVM) software includes a feature called Dynamic Multipathing (DMP), which provides disk multipathing as well as disk load balancing to optimize I/O throughput.

Sun StorEdge Traffic Manager is an architecture fully integrated within the Solaris OS (beginning with the Solaris 8 release) that enables I/O devices to be accessed through multiple host controller interfaces from a single instance of the I/O device.

For More Information on Multipathing Software

For instructions on how to configure and administer Solaris IP Network Multipathing, consult the IP Network Multipathing Administration Guide provided with your specific Solaris release.

For information about VVM and its DMP feature, refer to the documentation provided with the VERITAS Volume Manager software.

For information about Sun StorEdge Traffic Manager, refer to your Solaris OS documentation.

Storing FRU Information

The setfru command enables you to store information on FRU PROMs. For example, you might store information identifying the server in which the FRUs have been installed.

To Store Information in Available FRU PROMs

At the sc> prompt type:

setfru -c data