C H A P T E R 7

Troubleshooting the Server and Restoring ILOM Defaults

This chapter introduces the diagnostic tools you can use to troubleshoot or monitor the performance of your server. It also includes information about how to restore the service processor (SP) password and serial connection defaults in Oracle Integrated Lights Out Manager (ILOM), as well as how to restore your ILOM SP firmware.

Note - Oracle integrated Lights Out Manager formerly was called Sun Integrated Lights Out Manager.

Troubleshooting the Server

Oracle Diagnostic Tools

Diagnostic Tool Documentation

Using the Preboot Menu Utility

Accessing the Preboot Menu

Restoring ILOM to Default Settings

Restoring ILOM Access to the Serial Console

Restoring the SP Firmware Image

Preboot Menu Command Summary

7.1 Troubleshooting the Server

The server and its accompanying software and firmware contain many diagnostic tools and features that can help you:

Isolate problems when a failure occurs on a replaceable unit (CRU)

Monitor the status of a functioning system

Exercise the system to disclose more subtle problems that might be hardware related

Oracle provides a suite of diagnostic tools, each of which has its own specific strengths and applications. For more information about which tool might be best to use for your purpose, and where to locate information about these tools, see:

Oracle Diagnostic Tools

Diagnostic Tool Documentation

7.1.1 Oracle Diagnostic Tools

Oracle provides a wide selection of diagnostic tools for use with your server. These tools range from the SunVTS trademark software, a comprehensive validation test suite, to log files in ILOM that might contain clues helpful in identifying the possible sources of a problem, and the fault management function in ILOM that enables you to identify a faulted component as soon as the fault occurs.

The diagnostic tools range from standalone software packages, to firmware-based tests like power-on self-test (POST), U-Boot tests, or Pc-Check tests, to hardware LEDs that tell you when the system components are operating.

TABLE 7-1 summarizes the variety of diagnostic tools that you can use when troubleshooting or monitoring your server.

TABLE 7-1 Summary of Oracle Diagnostic Tools
Diagnostic Tool	Type	What It Does	Accessibility	Remote Capability
Integrated Lights Out Manager (ILOM)	SP firmware	Monitors environmental conditions, generates alerts, performs fault isolation, and provides remote console access.	Can function on standby power and when the operating system is not running.	Designed for remote and local access.
Preboot Menu	SP firmware	Enables you to restore some of ILOM defaults (including firmware) when ILOM is not accessible.	Can function on standby power and when operating system is not running.	Local access, but can be accessed remotely from a console running terminal emulation software.
LEDs	Hardware and SP firmware	Indicate status of overall system and particular components.	Available any time when system power is available.	Local, but sensor and indicators are accessible from ILOM web interface or command-line interface (CLI).
POST	Host firmware	Tests core components of system: CPUs, memory, and motherboard I/O bridge integrated circuits.	Runs on startup. Available when the operating system is not running.	Local, but can be accessed through ILOM Remote Console.
U-Boot	SP firmware	Initializes and test aspects of the service processor (SP) prior to booting the ILOM SP operating system. Tests SP memory, SP, network devices and I/O devices.	Can function on standby power and when operating system is not running.	Local or remote access through a serial connection.
Pc-Check	SP firmware	DOS-based utility that tests all motherboard components (CPU, memory, and I/O), ports, and slots.	Can function on standby power and when operating system is not running.	Remote access through Oracle ILOM Remote Console.
Oracle Solaris commands	Operating system software	Displays various kinds of system information.	Display various kinds of system information.	Local, and over network
SunVTS	Diagnostic tool standalone software	Exercises and stresses the system, running tests in parallel.	Requires operating system. Install SunVTS software separately.	View and control over network

7.1.2 Diagnostic Tool Documentation

TABLE 7-2 identifies where you can find more information about Oracle diagnostic tools.

TABLE 7-2 Summary of Documentation for Oracle Diagnostic Tools
Diagnostic Tool	Where to Find Information
ILOM	Oracle ILOM 3.0 Documentation Collection. Oracle Integrated Lights Out Manager (ILOM) 3.0 Supplement for Sun Fire X4170 M2 and X4270 M2 Servers Locate the latest version of these guides at: `http://docs.sun.com/app/docs/prod/sf.x4270m2#hic`
Preboot Menu	Using the Preboot Menu Utility
LEDs; or System indicators and sensors	Server Front Panel Features Server Back Panel Features Oracle ILOM 3.0 Documentation Collection. Oracle Integrated Lights Out Manager (ILOM) 3.0 Supplement for Sun Fire X4170 M2 and X4270 M2 Servers Locate the latest version of these guides at: `http://docs.sun.com/app/docs/prod/sf.x4270m2#hic`
POST	Default BIOS Power-On Self-Test (POST) Events BIOS POST F1 and F2 Errors
U-Boot or Pc-Check	Oracle x64 Servers Diagnostics Guide Locate the latest copy of this guide at: `http://docs.sun.com/app/docs/prod/sf.x4270m2#hic`
Oracle Solaris commands	Locate the latest Oracle Solaris command information for Solaris 10 at: `http://docs.sun.com/app/docs/doc/817-0550/eqxqt?a=view`
SunVTS	Download the SunVTS software at: `http://sunsolve.com` Locate the latest documentation for SunVTS at: `http://www.sun.com/oem/products/vts/index.html`

7.2 Using the Preboot Menu Utility

You can use the Preboot Menu utility installed on your server to make changes to some of the ILOM settings while ILOM is not currently running. The Preboot Menu also enables you to reset ILOM to its default settings, as well as restore the SP firmware image on your system.

For more information about how to use the Preboot Menu Utility, see these sections:

Accessing the Preboot Menu

Restoring ILOM to Default Settings

Restoring ILOM Access to the Serial Console

Restoring the SP Firmware Image

Preboot Menu Command Summary

7.2.1 Accessing the Preboot Menu

To access the Preboot Menu, you must reset the SP and interrupt the boot process. You can interrupt the ILOM boot process by either:

Using the Locate button on the local server.

For details about the requirements for this local access method, see Prerequisites for Accessing the Preboot Menu.

Typing xyzzy during a pause in the ILOM boot process from a remote serial terminal.

For details about the requirements for this remote access method, see Prerequisites for Accessing the Preboot Menu.

Some Preboot Menu settings must be configured first, and until they are, you must use the Locate button method to access the Preboot Menu.

For detailed instructions for accessing the Preboot Menu from a local or remote connection, see the following sections:

Prerequisites for Accessing the Preboot Menu

Access the Preboot Menu

Edit Preboot Menu for Remote Serial Access

7.2.1.1 Prerequisites for Accessing the Preboot Menu

Ensure that the applicable requirements are met prior to accessing the Preboot Menu from either a local or remote connection.

For local access to the Preboot Menu (using Locate button)

You must connect a terminal or computer running terminal emulation software to the server.

For more information about how to attach local devices to the server, see the Sun Fire X4170 M2 and X4270 M2 Servers Installation Guide (821-0481).

For instructions for accessing the Preboot Menu by using the Locate button, see Access the Preboot Menu.

For remote access to the Preboot Menu (typing xyzzy during the ILOM boot process)

Prior to accessing the Preboot Menu remotely, you must set the bootdelay and check_physical_presence settings in the Preboot Menu to enable remote access. To configure these settings for the first time, you need to:

a. Access the Preboot Menu using the Locate button on the local server as described in Access the Preboot Menu.

b. Edit the settings in the Preboot Menu to enable remote access as described in Edit Preboot Menu for Remote Serial Access.

c. Use a remote serial terminal or console running terminal emulation software to access the Preboot Menu remotely.

Note - You cannot use an SSH, or a remote KVMS session to access the Preboot Menu remotely.

7.2.1.2 Access the Preboot Menu

1. Ensure that the requirements in Prerequisites for Accessing the Preboot Menu are met.

2. Reset ILOM.

For example:

From the server SP ILOM, enter the command:

-> reset /SP

ILOM reboots and messages begin scrolling on the screen.

3. To interrupt the ILOM boot process, perform one of the following actions when the SP is resetting:

Press and hold the Locate button on the server front panel, until the Preboot Menu appears.

Type in xyzzy when you see the message:

Booting linux in n seconds...

Note - You cannot interrupt the ILOM boot process by typing xyzzy until you have configured the settings as described in Edit Preboot Menu for Remote Serial Access.

The ILOM Preboot Menu appears.

Booting linux in 10 seconds... 
 
                        ILOM Pre-boot Menu 
                        ------------------ 
Type "h" and [Enter] for a list of commands, or "?" [Enter] for 
command-line key bindings.  Type "h cmd" for summary of 'cmd' command. 
 
Warning: SP will warm-reset after 300 seconds of idle time. 
  Set 'bootretry' to -1 to disable the time-out. 
 
Preboot>

4. You can perform any of the following tasks or type boot to exit the Preboot Menu.

Enable remote access to the Preboot Menu.

For instructions, see Edit Preboot Menu for Remote Serial Access.

Restore ILOM settings to their defaults.

For instructions, see Resetting ILOM Configuration Using the Preboot Menu.

Restore the serial connection between ILOM and a host console.

For instructions, see Restore Access to the Serial Console Using the Preboot Menu.

Restore the SP firmware image.

For instructions, see Restore the SP Firmware Image Using the Preboot Menu.

Review the commands supported in the Preboot Menu.

For command details, see Preboot Menu Command Summary.

7.2.1.3 Edit Preboot Menu for Remote Serial Access

1. Access the Preboot Menu as described in Access the Preboot Menu.

2. At the Preboot> prompt, type edit.

The Preboot Menu enters edit mode.

In edit mode, the Preboot Menu displays its selections one-by-one, offering you a chance to change each one.

To change a setting, type the new value, then press Enter.

To skip to the next setting, press Enter.

3. Press Enter to move through the settings until the bootdelay setting appears.

4. To change the bootdelay setting, type 3, 10, or 30, then press Enter.

This value (3, 10, or 30) specifies the number of seconds the SP boot process delays while waiting for your input.

The Preboot Menu redisplays the bootdelay setting with the new value.

5. Press Enter to return to Preboot Menu selections.

The Preboot Menu selections appear.

6. Press Enter to move through the settings until the check_physical_presence setting appears.

To change the check_physical_presence setting, type no, then press Enter.

The Preboot Menu displays the check_physical_presence setting with the new value.

7. Press Enter for the new value to take effect.

The Preboot Menu asks you to confirm your changes.

Enter ‘y[es]’ to commit changes: [no]

8. Type y to save your changes and exit the edit session.

If you want to exit the edit session without saving your changes, type n.

9. To exit the Preboot Menu, type boot.

The following example shows an edit session where the check_physical_presence settings are changed.

Note - For a list of other settings you can edit in the Preboot Menu, see Edit Mode Settings in Preboot Menu.

Preboot> edit
Press Enter by itself to reach the next question.
Press control-C to discard changes and quit.
Values for baudrate are {[ 9600 ]| 19200 | 38400 | 57600 | 115200 }.
 Set baudrate?                [9600]
Values for serial_is_host are {[ 0 ]| 1 }.
 Set serial_is_host?          [0]
Values for bootdelay are { -1 | 3 | 10 | 30 }.
 Set bootdelay?               [3] 10
 Set bootdelay?               [10]
Values for bootretry are { -1 | 30 | 300 | 3000 }.
 Set bootretry?               [-1]
Values for diags_mode are {[ Normal ]| Quick | Extended | Skip }.
 Set diags_mode?              [Normal]
Values for preferred are {[ 0 ]| 1 }.
 Set preferred?               [<not set>]
Values for preserve_conf are {[ yes ]| no }.
 Set preserve_conf?           [yes]
Values for check_physical_presence are {[ yes ]| no }.
 Set check_physical_presence? [yes] no
 Set check_physical_presence? [no]
 Enter ’y[es]’ to commit changes: [no] y
Summary: Changed 2 settings.
Preboot>

7.2.1.4 Edit Mode Settings in Preboot Menu

In addition to changing the settings required in the Preboot Menu to enable remote access, you can also change other edit mode settings in the Preboot Menu. For a list of these settings, see TABLE 7-3.

TABLE 7-3 Edit Mode Preboot Menu Command Settings
Setting	Description
`baudrate`	Sets the baudrate of the serial port. Selections include 9600,19200, 38400, 57600, and 115200.
`serial_is_host`	If this is set to 0, the serial port connects to ILOM. If this is set to 1, the serial port connects to the host. For more details, see Restoring ILOM Access to the Serial Console.
`bootdelay`	The number of seconds the ILOM boot process waits for the user to enter `xyzzy` before booting the SP. Selections include `3`, `10`, or `30`.
`bootretry`	The number of seconds the Preboot Menu waits for user input before timing out and booting the SP. Set to -1 to disable the timeout.
`diags_mode`	Sets the diagnostic level executed when the server powers on.
`preferred`	Unused
`preserve_conf`	Setting this to `no` duplicates the function of the `unconfig ilom_conf` command, which resets many ILOM configuration settings.
`check_physical_presence`	If this is set to `Yes`, you must press and hold the Locate button to interrupt the SP boot process. If it is set to `No`, the boot process prompts you to interrupt it. See Edit Preboot Menu for Remote Serial Access for details.

7.2.2 Restoring ILOM to Default Settings

You can use the Preboot Menu to restore ILOM settings to their defaults.

7.2.2.1 Resetting ILOM Configuration Using the Preboot Menu

1. Access the Preboot Menu as described in Access the Preboot Menu.

2. At Preboot Menu prompt, type:

Preboot> unconfig ilom_conf

Setting ‘preserve_conf’ to ‘no’ for the next boot of ILOM.

3. Reset the SP by typing:

Preboot> boot

The Preboot Menu exits and the SP restarts.

After restarting the ILOM SP, the value for ILOM settings revert to their defaults.

7.2.3 Restoring ILOM Access to the Serial Console

In the event that the serial connection between ILOM and a host becomes unavailable, you can restore access to the serial port connection by reconfiguring the host as the external serial port owner in either the ILOM web interface or CLI, or in the Preboot Menu.

To determine which interface is best to use when restoring the serial connection between ILOM and a host console, consider the following:

If a network connection is available, you should use the ILOM web interface or CLI to restore ILOM’s access to the serial console.

For instructions, see the ILOM 3.0 Documentation Collection.

If a network connection is unavailable, you should use the procedure in Restore Access to the Serial Console Using the Preboot Menu to restore ILOM access to the serial console.

7.2.3.1 Restore Access to the Serial Console Using the Preboot Menu

1. Access the Preboot Menu as described in Accessing the Preboot Menu.

2. At the Preboot> prompt, type edit.

The Preboot Menu enters edit mode.

In edit mode, the Preboot Menu displays its selections one-by-one, offering you a chance to change each one.

To change a setting, type the new value, then press Enter.

To skip to the next setting, press Enter.

3. Press Enter to move through the settings until the serial_is_host setting appears.

To change the serial_is_host setting, type 0, and then press Enter.

The Preboot Menu appears with the new value for the serial_is_host setting.

4. Press Enter to display the Preboot Menu selections.

The Preboot Menu settings appear.

5. Press Enter to scroll through the settings until the Preboot Menu asks you to confirm your changes.

Enter ‘y[es]’ to commit changes: [no]

6. Type y to confirm your change and exit the edit session.

The Preboot Menu displays this message:

Summary: Changed 1 settings

Preboot>

7. To exit the Preboot Menu, type:

boot

7.2.4 Restoring the SP Firmware Image

If ILOM is available, you should always use the ILOM web interface or CLI to restore (update) the firmware image. For instructions about how to restore the SP firmware image using either the ILOM web interface or CLI, refer to the ILOM 3.0 Documentation Collection. If ILOM is unavailable, you can use the Preboot Menu or IPMIflash to restore the ILOM firmware image.

Note - If you are unable to access ILOM to update the SP firmware image using either the ILOM interfaces or IPMIflash, you should contact a service representative for assistance.

Caution - To restore the SP firmware image using the Preboot Menu, you must be a qualified service technician.

To use the Preboot Menu to restore the SP firmware image on the server, see the following sections:

Prerequisite for Restoring SP Firmware Using Preboot Menu

Restore the SP Firmware Image Using the Preboot Menu

7.2.4.1 Prerequisite for Restoring SP Firmware Using Preboot Menu

The following requirements must be met prior to restoring the SP firmware on your server using the Preboot Menu.

You must be a qualified service technician to perform this procedure.

You must have a valid .flash firmware image file on a TFTP server.

Note - Restore the SP firmware using the Preboot Menu requires a .flash file instead of a.pkg file that is typically used to update the SP using the ILOM interfaces.

7.2.4.2 Restore the SP Firmware Image Using the Preboot Menu

1. Access the Preboot Menu as described in Accessing the Preboot Menu.

Caution - The use of the netflashcommand is reserved for use by service personnel only.

2. At the Preboot> prompt, type:

net flash IPaddress path/name.flash

Where:

IPaddress is the IP address of a TFTP server

path is the path to the file relative to /tftpboot

name is the first part of the .flash file file name

For example:

net flash 10.8.173.25 images/system-rom.flash

After a series of messages, the Preboot Menu prompt appears.

Preboot>

3. At the Preboot> prompt, type the reset command to restart the SP.

For example:

Preboot> reset

The Preboot Menu exits and ILOM restarts.

7.2.5 Preboot Menu Command Summary

TABLE 7-4 identifies the Preboot Menu commands.

TABLE 7-4 Preboot Menu Commands
Command	Description
`boot`	Boots ILOM. The Preboot Menu exits and ILOM restarts. Note - This command executes a modified boot sequence that does not offer the choice to select the diagnostic level, or to interrupt the boot sequence and return to the Preboot Menu. To execute the normal boot sequence, use the `reset warm` command instead.
`vers`	Displays version information including the hardware type, board revision, ILOM revision, revisions of Preboot software (PBSW) and recovery U-Boot. Shows the checksum integrity of the images, and the preference between redundant images.
help	Displays a list of commands and parameters.
`show`	Displays a list of SP settings.
`edit`	Starts an interactive dialog that prompts and changes settings one-by-one. See Edit Preboot Menu for Remote Serial Access for details.
`diag`	Runs the U-Boot diagnostic tests in manual mode. See the Oracle X64 Servers Diagnostics Guide (820-6750) for more on U-Boot diagnostic tests.
`host`	Initiates various activities related to the host. `clearcmos` - Clears CMOS and BIOS passwords. `console` - Connects SP console to host serial console. Note - Type `Ctrl \ q` to quit. `show` - Shows information about the host state. `enable-on` - Enables the front-panel Power button, which is usually disabled unless ILOM is running. Caution - If you start the host when ILOM is off, the BIOS does not send error events, or power messages to the SP. This can cause the server to lose power. `hard-off` - Turns the host off.
`net`	`{ config \| dhcp \| ping \| flash }` `config` - Starts a dialog that enables you to change the ILOM network settings. `dhcp` - Changes the network addressing from static to DHCP. Note - You must set `ipdiscovery` `=` `dhcp` using the `net` `config` command first. `ping` - Sends a ping. `flash` - Downloads an ILOM firmware image. See Restoring the SP Firmware Image for details. Type the `help net` command for more details on these commands.
`reset`	{ `warm \| cold` } `warm` - Resets the SP without affecting a running host. `cold` - Resets the SP and the host. It has the effect of powering off the server.
`unconfig`	`{ ilom_conf \| most \| all }` Causes ILOM to erase any configuration information and returns the values to defaults the next time ILOM boots. `ilom_conf` - Resets configuration settings but preserves SP network and `baudrate`, `preferred`, and `check_physical_presence`. `most`- Resets the SP data storage, but preserves SP `network` and `baudrate`, `preferred`, and `check_physical_presence` settings. `all` - Resets all SP data storage and settings. Booting ILOM restores other defaults. Note - None of these options erases the dynamic FRU PROMs.