C H A P T E R 8

Troubleshooting The Server and Restoring ILOM Defaults

This chapter introduces the diagnostic tools you can use to troubleshoot or monitor the performance of your server. It also includes information about how to restore the SP password and serial connection defaults in ILOM, as well as how to restore your ILOM SP firmware.

Troubleshooting The Server

Sun Diagnostic Tools

Diagnostic Tool Documentation

Restoring ILOM Defaults

Accessing the Preboot Menu

Restoring the Factory Default ILOM Root Password

Restoring ILOM Access to the Serial Console

Restoring the SP Firmware Image

Preboot Menu Command Summary

8.1 Troubleshooting The Server

The server and its accompanying software and firmware contain many diagnostic tools and features that can help you:

Isolate problems when a failure occurs on a customer replaceable unit (CRU)

Monitor the status of a functioning system

Exercise the system to disclose more subtle problems that might be hardware related

Sun provides a suite of diagnostic tools, each of which has its own specific strengths and applications. For more information about which tool might be best to use for your purpose, and where to locate information about these tools, see:

Sun Diagnostic Tools

Diagnostic Tool Documentation

8.1.1 Sun Diagnostic Tools

Sun provides a wide selection of diagnostic tools for use with your server. These tools range from the SunVTS software, a comprehensive validation test suite, to log files in ILOM that might contain clues helpful in identifying the possible sources of a problem, and the fault management function in ILOM that enables you to identify a faulted component as soon as the fault occurs.

The diagnostic tools range from standalone software packages, to firmware-based tests like power-on self-test (POST), U-Boot tests, or Pc-Check tests, to hardware LEDs that tell you when the system components are operating.

TABLE 8-1 summarizes the variety of diagnostic tools that you can use when troubleshooting or monitoring your server.

TABLE 8-1 Summary of Sun Diagnostic Tools
Diagnostic Tool	Type	What It Does	Accessibility	Remote Capability
Integrated Lights Out Manager (ILOM)	SP firmware	Monitors environmental conditions, generates alerts, performs fault isolation, and provides remote console access.	Can function on standby power and when the operating system is not running.	Designed for remote and local access.
Preboot Menu	SP firmware	Enables you to restore some of ILOM defaults (including firmware) when ILOM is not accessible.	Can function on standby power and when operating system is not running.	Local access, but can be accessed remotely from a console running terminal emulation software.
LEDs	Hardware and SP firmware	Indicate status of overall system and particular components.	Available any time on system power is available.	Local, but sensor and indicators are accessible from ILOM web interface or command-line interface (CLI).
POST	Host firmware	Tests core components of system: CPUs, memory, and motherboard I/O bridge integrated circuits.	Runs on startup. Available when the operating system is not running.	Local, but can be accessed through ILOM Remote Console.
U-Boot	SP firmware	Initializes and test aspects of the service processor (SP) prior to booting the ILOM SP operating system. Tests SP memory, SP, network devices and I/O devices.	Can function on standby power and when operating system is not running.	Local or remote access through a serial connection.
Pc-Check	SP firmware	DOS-based utility that tests all motherboard components (CPU, memory, and I/O), ports, and slots.	Can function on standby power and when operating system is not running.	Remote access through Sun ILOM Remote Console.
Solaris commands	Operating system software	Displays various kinds of system information.	Display various kinds of system information.	Local, and over network
SunVTS	Diagnostic tool standalone software	Exercises and stresses the system, running tests in parallel.	Requires operating system. Install SunVTS software separately.	View and control over network

8.1.2 Diagnostic Tool Documentation

TABLE 8-2 identifies where you can find more information about Sun diagnostic tools.

TABLE 8-2 Summary of Documentation for Sun Diagnostic Tools
Diagnostic Tool	Where to Find Information
ILOM	Sun Integrated Lights Out Manager 2.0 User’s Guide. Addendum to the Sun Integrated Lights Out Manager 2.0 User’s Guide Sun Integrated Lights Out Manager (ILOM) 2.0 Supplement for Sun Fire X4170, X4270, and X4275 Servers Locate the latest version of these guides at: `http://docs.sun.com/app/docs/prod/sf.x4170#hic`
Preboot Menu	Restoring ILOM Defaults.
LEDs; or System indicators and sensors	Sun Fire X4170, X4270, and X4275 Servers Service Manual Sun Integrated Lights Out Manager 2.0 User’s Guide Sun Integrated Lights Out Manager (ILOM) 2.0 Supplement for Sun Fire X4170, X4270, and X4275 Servers Locate the latest version of these guides at: `http://docs.sun.com/app/docs/prod/sf.x4170#hic`
POST	Default BIOS Power-On Self-Test (POST) Events BIOS POST F1 and F2 Errors
U-Boot or Pc-Check	Sun x64 Servers Diagnostics Guide Locate the latest copy of this guide at: `http://docs.sun.com/app/docs/prod/sf.x4170#hic`
Solaris commands	Locate the latest Solaris command information for Solaris 10 at: `http://docs.sun.com/app/docs/doc/817-0550/eqxqt?a=view`
Sun VTS	Download the Sun VTS software at: `http://sunsolve.com` Locate the latest documentation for SunVTS at: `http://www.sun.com/oem/products/vts/index.html`

8.2 Restoring ILOM Defaults

You can restore the factory defaults for the following ILOM features:

ILOM root password

Serial port connection

To restore these factory defaults, you must use the Preboot Menu utility that is shipped installed on your server. The Preboot Menu enables you to address changes to some of ILOM’s settings while ILOM is not currently running. In addition to restoring factory defaults for the root password and serial connection settings, the Preboot Menu enables you to restore the SP firmware image on your system.

For more information about how to use the Preboot Menu to restore settings in ILOM, see these sections:

Accessing the Preboot Menu

Restoring the Factory Default ILOM Root Password

Restoring ILOM Access to the Serial Console

Restoring the SP Firmware Image

Preboot Menu Command Summary

8.2.1 Accessing the Preboot Menu

To access the Preboot Menu, you must reset the SP and interrupt the boot process. You can interrupt the ILOM boot process by either:

Using the Locate button on the local server.

For details about the requirements for this local access method, see Prerequisites for Accessing the Preboot Menu

Typing xyzzy during a pause in the bootstrap process from a remote console (terminal or computer).

For details about the requirements for this remote access method, see Prerequisites for Accessing the Preboot Menu.

Some Preboot Menu settings must be configured first, and until they are, you must use the Locate button method to access the Preboot Menu.

For detailed instructions for accessing the Preboot Menu from a local or remote connection, see the following sections:

Prerequisites for Accessing the Preboot Menu

Access to the Preboot Menu

Edit Preboot Menu for Remote Access

8.2.1.1 Prerequisites for Accessing the Preboot Menu

Ensure that the applicable requirements are met prior to accessing the Preboot menu from either a local or remote connection.

For local access to the Preboot Menu (using Locate button)

You must connect a terminal or computer running terminal emulation software to the server.

For more information about how to attach local devices to the server using a dongle cable, see the Sun Fire X4170, X4270, and X4275 Servers Installation Guide.

For instructions for accessing the Preboot Menu by using the Locate button, see Access to the Preboot Menu.

For remote access to the Preboot Menu (typing xyzzy during bootstrap)

Prior to accessing the Preboot Menu remotely, you must set the bootdelay and check_physical_presence settings in the Preboot Menu to enable remote access. To configure these settings for the first time, you need to:

a. Access the Preboot Menu using the Locate button on the local server as described in Access to the Preboot Menu.

b. Edit the settings in the Preboot Menu to enable remote access as described in Edit Preboot Menu for Remote Access.

c. Use a remote terminal or computer running terminal emulation software to access the Preboot Menu remotely.

Note - You cannot use an SSH, or a remote KVMS session to access the Preboot Menu remotely.

8.2.1.2 Access to the Preboot Menu

1. Ensure that the requirements in Prerequisites for Accessing the Preboot Menu are met.

2. Reset ILOM.

For example:

From the server SP ILOM, enter the command:

-> reset /SP

Where n is the slot number of the node.

Temporarily remove power from the server by disconnecting and reconnecting the power cord(s).

ILOM reboots and messages begin scrolling on the screen.

3. To interrupt the ILOM boot process perform one of the following actions when the SP is resetting:

Press and hold the Locate button on the server front panel, until the Preboot Menu appears.

Type in xyzzy when you see the message:

Booting linux in n seconds...

Note - You cannot interrupt the ILOM boot process by typing xyzzy until you have configured the settings as described in Edit Preboot Menu for Remote Access.

The ILOM Preboot Menu appears.

Booting linux in 10 seconds... 
 
                        ILOM Pre-boot Menu 
                        ------------------ 
Type "h" and [Enter] for a list of commands, or "?" [Enter] for 
command-line key bindings.  Type "h cmd" for summary of 'cmd' command. 
 
Warning: SP will warm-reset after 300 seconds of idle time. 
  Set 'bootretry' to -1 to disable the time-out. 
 
Preboot>

4. You can perform any of the following tasks or type boot to exit the Preboot Menu.

Enable remote access to the Preboot Menu.

For instructions, see Edit Preboot Menu for Remote Access.

Restore the factory default to the ILOM root password.

For instructions, see Restore ILOM Root Password to Factory Default Using the Preboot Menu.

Restore the serial connection between ILOM and a host console.

For instructions, see Restore Access to the Serial Console Using the Preboot Menu.

Restore the SP firmware image.

For instructions, see Restore the SP Firmware Image Using the Preboot Menu.

Review the commands supported in the Preboot Menu.

For command details, see Preboot Menu Command Summary.

8.2.1.3 Edit Preboot Menu for Remote Access

1. Access the Preboot Menu as described in Access to the Preboot Menu.

2. At the Preboot> prompt, type edit.

The Preboot Menu enters edit mode.

In edit mode, the Preboot Menu displays its selections one-by-one, offering you a chance to change each one.

To change a setting, type the new value, then press Enter.

To skip to the next setting, press Enter.

3. Press Enter to move through the settings until the bootdelay setting appears.

4. To change the bootdelay setting, type 3, 10, or 30, then press Enter.

This value (3, 10, or 30) specifies the number of seconds the SP boot process delays while waiting for your input.

The Preboot Menu re-displays the bootdelay setting with the new value.

5. Press Enter to return to Preboot Menu selections.

The Preboot Menu selections appear.

6. Press Enter to move through the settings until the check_physical_presence setting appears.

To change the check_physical_presence setting, type no, then press Enter.

The Preboot Menu displays the check_physical_presence setting with the new value.

7. Press Enter for the new value to take effect.

The Preboot Menu asks you to confirm your changes.

Enter ‘y[es]’ to commit changes: [no]

8. Type y to save your changes and exit the edit session.

If you want to exit the edit session without saving your changes, type n.

The following example shows an edit session where the bootdelay and check_physical_presence settings are changed.

Note - For a list of other settings you can edit in the Preboot Menu, see Edit Mode Settings in Preboot Menu..

Preboot> edit
 
Press Enter by itself to reach the next question.
  Press control-C to discard changes and quit.
 
 Values for baudrate are {[ 9600 ]| 19200 | 38400 | 57600 | 115200 }.
  Set baudrate?                [9600]
 Values for serial_is_host are {[ 0 ]| 1 }.
  Set serial_is_host?          [0]
 Values for bootdelay are { -1 | 3 | 10 | 30 }.
  Set bootdelay?               [30] 10
  Set bootdelay?               [10]
 Values for bootretry are { -1 | 30 | 300 | 3000 }.
  Set bootretry?               [<not set>]
 Values for preferred are {[ 0 ]| 1 }.
  Set preferred?               [<not set>]
 Values for preserve_conf are {[ yes ]| no }.
  Set preserve_conf?           [yes]
 Values for preserve_users are {[ yes ]| no }.
  Set preserve_users?          [no]
 Values for preserve_password are {[ yes ]| no }.
  Set preserve_password?       [yes]
 Values for check_physical_presence are {[ yes ]| no }.
  Set check_physical_presence? [no] no
  Set check_physical_presence? [no]
 Enter 'y[es]' to commit changes: [no] y
Summary: Changed 2 settings.
Preboot>

8.2.1.4 Edit Mode Settings in Preboot Menu.

In addition to changing the settings required in the Preboot Menu to enable remote access, you can also change other edit mode settings in the Preboot Menu. For a list of these settings, see TABLE 8-3:

TABLE 8-3 Edit Mode Preboot Menu Command Settings
Setting	Description
`baudrate`	Sets the baudrate of the serial port. Selections include 9600,19200, 38400, 57600, and 115200.
`serial_is_host`	If this is set to 0, the serial port connects to the ILOM. If this is set to 1, the serial port connects to the host. For more details, see Restoring ILOM Access to the Serial Console.
`bootdelay`	The number of seconds the bootstrap process waits for the user to enter `xyzzy` before booting the SP.
`bootretry`	The number of seconds the Preboot Menu waits for user input before timing out and starting the SP. Set to -1 to disable the timeout.
`preferred`	Unused
`preserve_conf`	Setting this to `no` duplicates the function of the `unconfig ilom_conf` command, which resets many ILOM configuration settings, but preserves SP network, `baudrate`, and `check_physical_presence` the next time the SP is booted.
`preserve_users`	Setting this to `no` duplicates the function of the `unconfig users` command, which resets user information to the default values the next time the SP is booted.
`preserve_ password`	Setting this to `no` duplicates the function of the `unconfig password` command, which resets the root password to the default value the next time the SP is booted.
`check_physical_presence`	If this is set to `Yes`, you must press and hold the Locate button to interrupt the SP boot process. If it is set to `No`, the boot process prompts you to interrupt it. See Edit Preboot Menu for Remote Access for details.

8.2.2 Restoring the Factory Default ILOM Root Password

The ILOM root password grants you access to the ILOM web or command line (CLI) interfaces on the SP. If you forget the root password, you can use the Preboot Menu to restore the password to the factory default, changeme.

8.2.2.1 Restore ILOM Root Password to Factory Default Using the Preboot Menu

1. Access the Preboot Menu as described in Accessing the Preboot Menu.

2. In Preboot Menu, type:

Preboot> unconfig password

Setting ‘preserve_password’ to ‘no’ for the next boot of ILOM.

3. Reset the SP by typing:

Preboot> boot

The Preboot Menu exits and the SP restarts.

After restarting the ILOM SP, the value for the root password (on the ILOM SP) is set to changeme when the SP is finished booting.

8.2.3 Restoring ILOM Access to the Serial Console

In the event that the serial connection between ILOM and a host becomes unavailable, you can restore access to the serial port connection by reconfiguring the host as the external serial port owner in either the ILOM web interface or CLI, or in the Preboot Menu.

To determine which interface is best to use when restoring the serial connection between ILOM and a host console, consider the following:

If a network connection is available, you should use the ILOM web interface or CLI to restore ILOM’s access to the serial console.

For instructions, see the procedure for “Switching Serial Port Output Between SP and Host Console” in the Sun Integrated Lights Out Manager (ILOM) 2.0 Supplement for Sun Fire X4170, X4270, and X4275 Servers.

If a network connection is unavailable, you should use the procedure in Restore Access to the Serial Console Using the Preboot Menu to restore ILOM access to the serial console.

8.2.3.1 Restore Access to the Serial Console Using the Preboot Menu

1. Access the Preboot Menu as described in Accessing the Preboot Menu.

2. At the Preboot> prompt, type edit.

The Preboot Menu enters edit mode.

In edit mode, the Preboot Menu displays its selections one-by-one, offering you a chance to change each one.

To change a setting, type the new value, then press Enter.

To skip to the next setting, press Enter.

3. Press Enter to move through the settings until the serial_is_host setting appears.

To change the serial_is_host setting, type 0, and then press Enter.

The Preboot Menu redisplays the serial_is_host setting with the new value.

4. Press Enter to display the Preboot Menu selections.

The Preboot Menu settings appear.

5. Press Enter to scroll through the settings until the Preboot Menu asks you to confirm your changes.

Enter ‘y[es]’ to commit changes: [no]

6. Type y to confirm your change and exit the edit session.

The preboot menu displays this message

Summary: Changed 1 settings

Preboot>

7. To exit the Preboot Menu, type: boot.

8.2.4 Restoring the SP Firmware Image

If ILOM is available, you should always use the ILOM web interface or CLI to restore (update) the firmware image. For instructions about how to restore the SP firmware image using either the ILOM web interface or CLI, see the Sun Integrated Lights Out Manager 2.0 User Guide (820-1188).

If ILOM is unavailable, you can use the Preboot Menu or IPMIflash to restore the ILOM firmware image.

To restore the SP firmware image using IPMIflash, see the Addendum to the Sun Integrated Lights Out Manager 2.0 User’s Guide (820-4198) for more for details.

Note - If you are unable to access ILOM to update the SP firmware image using either the ILOM interfaces or IPMIflash, you should contact a Sun service representative for assistance.

To use the Preboot Menu to restore the SP firmware image on the server, see the following sections:

Prerequisite for Restoring SP Firmware Using Preboot Menu

Special Recovery Considerations for Systems Running ILOM Firmware 2.0.2.17 or Later

Restore the SP Firmware Image Using the Preboot Menu

8.2.4.1 Prerequisite for Restoring SP Firmware Using Preboot Menu

The following requirements must be met prior to restoring the SP firmware on your server using the Preboot Menu.

If your system is running ILOM firmware 2.0.2.x or later, you must enable the Preboot Menu to support the ability to restore the SP firmware . For more infomation on how to enable support in the Preboot Menu to recover the SP firmware, see Special Recovery Considerations for Systems Running ILOM Firmware 2.0.2.17 or Later.

You must be a Sun qualified service technician to perform this procedure.

You must have a valid .flash firmware image file on a TFTP server.

Note - Restore the SP firmware using the Preboot Menu requires a .flash file instead of a.pkg file that is typically used to update the SP using the ILOM interfaces.

8.2.4.2 Special Recovery Considerations for Systems Running ILOM Firmware 2.0.2.17 or Later

As of ILOM 2.0.2.17, you must enable support in the Preboot Menu to recover the SP firmware image prior to performing the steps described in Restore the SP Firmware Image Using the Preboot Menu.

To enable support in the Preboot Menu to recover ILOM firmware 2.0.2.17 or later, follow these steps.

1. Prepare the server for service by powering down the server, extending the server to the maintenance position, and removing the top cover from the server.

For instructions, see the following sections:

Powering On and Off the Server

Extending the Server to the Maintenance Position

Removing the Top Cover

2. Place a jumper on J602 to short pins 2 and 3 (see FIGURE 8-1).

FIGURE 8-1 J602 Jumper Location

Graphic showing the location of the J602 Jumper.

3. Replace the top cover and power on the server.

For instructions, see the following sections:

Install Top Cover

Powering On the Server

4. Follow the instructions for restoring the SP firmware using the Preboot Menu in Restore the SP Firmware Image Using the Preboot Menu and proceed to the next step in this procedure.

Note - The Preboot Menu firmware recovery process must be performed by a Sun qualified service technician and you must have a valid .flash file to perform the procedure.

5. After you restore the SP image using the Preboot menu, perform the following steps to remove the J602 jumper from the server and to return the server to normal operation.

a. Power off the server.

See Powering On and Off the Server.

b. Remove the top cover from the server.

See Removing the Top Cover.

c. Remove the jumper from J602.

d. Replace the top cover.

See Install Top Cover.

e. Return the server to the normal rack position.

See Returning the Server to the Normal Rack Position.

f. Power on the server.

See Powering On the Server.

8.2.4.3 Restore the SP Firmware Image Using the Preboot Menu

1. Access the Preboot Menu as described in Accessing the Preboot Menu.

2. At the Preboot> prompt, type:

Caution - The use of the net flash command is reserved for use by Sun service personnel only.

net flash IPaddress path/name.flash

where:

IPaddress is the IP address of a TFTP server

path is the path to the file relative to /tftpboot

name is the first part of the .flash file filename

For example:

Preboot> net flash 10.8.173.25 images/system-rom.flash

After a series of messages, the Preboot Menu prompt appears.

Preboot>

3. At the Preboot> prompt, type the reset command to restart the SP.

For example:

Preboot> reset

The Preboot Menu exits and ILOM restarts.

8.2.5 Preboot Menu Command Summary

TABLE 8-4 identifies the Preboot Menu commands.

TABLE 8-4 Preboot Menu Commands
Command	Description
`boot`	Boots ILOM. The Preboot Menu exits and ILOM restarts. Note - This command executes a modified boot sequence that does not offer the choice to select the diagnostic level, or to interrupt the boot sequence and return to the Preboot Menu. To execute the normal boot sequence, use the `reset warm` command instead.
`vers`	Displays version information including the hardware type, board revision, ILOM revision, revisions of PBSW and recovery U-Boot. Shows the checksum integrity of the images, and the preference between redundant images.
help	Displays a list of commands and parameters.
`show`	Displays a list of SP settings.
`edit`	Starts an interactive dialog that prompts and changes settings one-by-one. See Edit Preboot Menu for Remote Access for details.
`diag`	Runs the U-Boot diagnostic tests in manual mode. See the Sun X64 Servers Diagnostics Guide for more on U-Boot diagnostic tests.
`host`	Initiates various activities related to the host. `clearcmos` - Clears CMOS and BIOS passwords. `console` - Connects SP console to host serial console. Note - Type `Ctrl \ q` to quit. `show` - Shows information about the host state. `enable-on` - Enables the front-panel Power button, which is usually disabled unless ILOM is running. Caution - If you start the host when ILOM is off, the BIOS does not send error events, or power messages to the SP. This can cause the server to lose power. `hard-off` - Turns the host off.
`net`	`{ config \| dhcp \| ping \| flash }` `config` - Starts a dialog that enables you to change the ILOM network settings. `dhcp` - Changes the network addressing from static to DHCP. Note - You must set `ipdiscovery` `=` `dhcp` using the `net` `config` command first. `ping` - Sends a ping. `flash` - Downloads an ILOM firmware image. See Restoring the SP Firmware Image for details. Type `help net` command for more details on these commands.
`reset`	{[ `warm ]\| cold` }. Resets the SP and the host. `warm` - Resets the SP without affecting a running host. `cold` - Resets the SP and the host. It has the effect of powering off the server.
`unconfig`	`{ users \| ilom_conf \| most \| all }` Causes ILOM to erase any configuration information and returns the values to defaults the next time ILOM boots. `users` - Resets all configured user information. `password` -Resets the ILOM root password to the default. See Restoring the Factory Default ILOM Root Password for more details. `ilom_conf` - Resets configuration settings but preserves SP network and `baudrate`, `preferred`, and `check_physical_presence`. `most` - Resets the SP data storage, but preserves network and baud rate, preferred, and `check_physical_presence` settings. `all` - Resets all SP data storage and settings. Booting ILOM restores other defaults. Note - None of these options erases the dynamic FRU PROMs.