C H A P T E R  4

 


Troubleshooting the Server and Restoring ILOM Defaults

This chapter introduces the diagnostic tools you can use to troubleshoot or monitor the performance of your server. It also includes information about how to restore the service processor (SP) password and serial connection defaults in ILOM, as well as how to restore your ILOM SP firmware.


4.1 Troubleshooting the Server

The server module and its accompanying software and firmware contain many diagnostic tools and features that can help you complete the following tasks:

A suite of diagnostic tools, each of which has its own specific strengths and applications is available for troubleshooting server issues. For more information about which tool might be best to use for your purpose, and where to locate information about these tools, see:

4.1.1 Diagnostic Tools

There is a wide selection of diagnostic tools for use with your server. These tools range from the SunVTS software, a comprehensive validation test suite, to log files in ILOM that might contain clues helpful in identifying the possible sources of a problem, and the fault management function in ILOM that enables you to identify a faulted component as soon as the fault occurs.

The diagnostic tools also include standalone software packages, firmware-based tests like power-on self-test (POST), U-Boot tests, or Pc-Check tests, and hardware LEDs that tell you when the system components are operating.

TABLE 4-1 summarizes the variety of diagnostic tools that you can use when troubleshooting or monitoring your server.


TABLE 4-1 Summary of Server Diagnostic Tools

Diagnostic Tool

Type

What It Does

Accessibility

Remote Capability

Integrated Lights Out Manager (ILOM)

SP firmware

Monitors environmental conditions, generates alerts, performs fault isolation, and provides remote console access.

Can function on standby power and when the operating system is not running.

Designed for remote and local access.

Preboot Menu

SP firmware

Enables you to restore some of ILOM defaults (including firmware) when ILOM is not accessible.

Can function on standby power and when operating system is not running.

Local, but remote serial access is possible if the SP serial port is connected to a network-accessible terminal server.

LEDs

Hardware and SP firmware

Indicate status of overall system and particular components.

Available when system power is available.

Note - Fault Remind LEDs are available for a few minutes after the power is removed.

Local, but sensor and indicators are accessible from ILOM web interface or command-line interface (CLI).

POST

Host firmware

Tests core components of system: CPUs, memory, and motherboard I/O bridge integrated circuits.

Runs on startup. Available when the operating system is not running.

Local, but can be accessed through ILOM Remote Console.

U-Boot

SP firmware

Initializes and tests aspects of the SP prior to booting the ILOM SP operating system. Tests SP memory, SP, network devices and I/O devices.

Can function on standby power and when operating system is not running.

Local, but remote serial access is possible if the SP serial port is connected to a network-accessible terminal server.

Pc-Check

SP firmware

(or on Tools and Driver DVD for systems without SP)

DOS-based utility that tests all motherboard components (CPU, memory, and I/O), ports, and slots.

Runs on startup. Available when the operating system is not running.

Remote access through ILOM Remote Console for servers shipped with SP.

or

Local access for servers shipped without an SP (via Tools and Driver DVD).

Solaris commands

Operating system software

Displays various kinds of system information.

Requires operating system.

Local, and over network.

SunVTS

Diagnostic tool standalone software

Exercises and stresses the system, running tests in parallel.

Requires operating system. Install SunVTS software separately.

View and control over network.


4.1.2 Diagnostic Tool Documentation

TABLE 4-2 identifies where you can find more information about Sun diagnostic tools.


TABLE 4-2 Summary of Documentation for Sun Diagnostic Tools

Diagnostic Tool

Where to Find Information

ILOM

  • Oracle Integrated Lights Out Manager (ILOM) 3.0 Documentation Collection
  • Oracle Integrated Lights Out Manager (ILOM) 3.0 Supplement for Sun Netra X6270 M2 Server Module, 821-0943


Locate the latest version of these guides at these URLs:

Preboot Menu

LEDs;

or

System indicators and sensors

  • This service manual
  • Oracle Integrated Lights Out Manager (ILOM) 3.0 Documentation Collection
  • Oracle Integrated Lights Out Manager (ILOM) 3.0 Supplement for Sun Netra X6270 M2 Server Module, 821-0943


Locate the latest version of these guides at:

POST

U-Boot

or

Pc-Check

  • Oracle x86 Servers Diagnostics Guide, 820-6750


Locate the latest verson of this guide at:

Solaris commands

Locate the latest Solaris command information for Solaris 10 at:

SunVTS

Locate the latest documentation for SunVTS at:



4.2 Restoring ILOM Defaults

You can restore the factory defaults for the following ILOM features:

To restore these factory defaults, you must use the Preboot Menu utility that is shipped installed on your server. The Preboot Menu enables you to address changes to some of ILOM’s settings while ILOM is not currently running. In addition to restoring factory defaults for the root password and serial connection settings, the Preboot Menu enables you to restore the SP firmware image on your system.

For more information about how to use the Preboot Menu to restore settings in ILOM, see these sections:

4.2.1 Accessing the Preboot Menu

To access the Preboot Menu, you must reset the SP and interrupt the boot process. You can interrupt the ILOM boot process by either:

For details about the requirements for this local access method, see Prerequisites for Accessing the Preboot Menu.

Or

For details about the requirements for this remote serial access method, see Prerequisites for Accessing the Preboot Menu.

Some Preboot Menu settings must be configured first, and until they are, you must use the Locate button method to access the Preboot Menu.

For detailed instructions for accessing the Preboot Menu from a local or remote connection, see the following sections:

4.2.1.1 Prerequisites for Accessing the Preboot Menu

Ensure that the applicable requirements are met prior to accessing the Preboot Menu from either a local or remote connection.

You must connect a terminal or console capable of running terminal emulation software to the serial port on the server using a dongle cable.

For more information about how to attach local devices to the server using a dongle cable, see Attaching Devices to the Server Module.

For instructions for accessing the Preboot Menu by using the Locate button, see Access to the Preboot Menu.

Prior to accessing the Preboot Menu remotely, you must set the bootdelay and check_physical_presence settings in the Preboot Menu to enable remote serial access. To configure these settings for the first time, you need to:

a. Access the Preboot Menu using the Locate button on the local server as described in Access to the Preboot Menu.

b. Edit the settings in the Preboot Menu to enable remote access as described in Edit Preboot Menu for Remote Serial Access.

c. Use a remote serial terminal or console running terminal emulation software to access the Preboot Menu remotely.



Note - You cannot use an SSH, or a remote KVMS session to access the Preboot Menu remotely.



procedure icon  Access to the Preboot Menu

1. Ensure that the requirements in Prerequisites for Accessing the Preboot Menu are met.

2. Reset ILOM.

For example:

-> reset /SP

-> reset /CH/BLn/SP

Where n is the slot number of the server module.

ILOM reboots and messages begin scrolling on the screen.

3. To interrupt the ILOM boot process, perform one of the following actions when the SP is resetting:

Or

Booting linux in n seconds...



Note - You cannot interrupt the ILOM boot process by typing xyzzy until you have configured the settings as described in Edit Preboot Menu for Remote Serial Access.


The ILOM Preboot Menu appears.


Booting linux in 10 seconds... 
 
                        ILOM Pre-boot Menu 
                        ------------------ 
Type "h" and [Enter] for a list of commands, or "?" [Enter] for 
command-line key bindings.  Type "h cmd" for summary of 'cmd' command. 
 
Warning: SP will warm-reset after 300 seconds of idle time. 
  Set 'bootretry' to -1 to disable the time-out. 
 
Preboot> 

4. You can perform any of the following tasks or type boot to exit the Preboot Menu.

For instructions, see Edit Preboot Menu for Remote Serial Access.

For instructions, see Restore ILOM Root Password to Factory Default Using the Preboot Menu.

For instructions, see Restore Access to the Serial Console Using the Preboot Menu.

For instructions, see Restore the SP Firmware Image Using the Preboot Menu.

For command details, see Preboot Menu Command Summary.


procedure icon  Edit Preboot Menu for Remote Serial Access

1. Access the Preboot Menu as described in Access to the Preboot Menu.

2. At the Preboot> prompt, type edit.

The Preboot Menu enters edit mode.

In edit mode, the Preboot Menu displays its selections one-by-one, offering you a chance to change each one.

3. Press Enter to move through the settings until the bootdelay setting appears.

4. To change the bootdelay setting, type 3, 10, or 30, then press Enter.

This value (3, 10, or 30) specifies the number of seconds the SP boot process delays while waiting for your input.

The Preboot Menu redisplays the bootdelay setting with the new value.

5. Press Enter to return to Preboot Menu selections.

The Preboot Menu selections appear.

6. Press Enter to move through the settings until the check_physical_presence setting appears.

To change the check_physical_presence setting, type no, then press Enter.

The Preboot Menu displays the check_physical_presence setting with the new value.

7. Press Enter for the new value to take effect.

The Preboot Menu asks you to confirm your changes.

Enter ‘y[es]’ to commit changes: [no]

8. Type y to save your changes and exit the edit session.

If you want to exit the edit session without saving your changes, type n.

The following example shows an edit session where the bootdelay and check_physical_presence settings are changed.



Note - For a list of other settings you can edit in the Preboot Menu, see Edit Mode Settings in Preboot Menu.



Preboot> edit
 
Press Enter by itself to reach the next question.
  Press control-C to discard changes and quit.
 
 Values for baudrate are {[ 9600 ]| 19200 | 38400 | 57600 | 115200 }.
  Set baudrate?                [9600]
 Values for serial_is_host are {[ 0 ]| 1 }.
  Set serial_is_host?          [0]
 Values for bootdelay are { -1 | 3 | 10 | 30 }.
  Set bootdelay?               [30] 10
  Set bootdelay?               [10]
 Values for bootretry are { -1 | 30 | 300 | 3000 }.
  Set bootretry?               [<not set>]
 Values for preferred are {[ 0 ]| 1 }.
  Set preferred?               [<not set>]
 Values for preserve_conf are {[ yes ]| no }.
  Set preserve_conf?           [yes]
 Values for preserve_users are {[ yes ]| no }.
  Set preserve_users?          [no]
 Values for preserve_password are {[ yes ]| no }.
  Set preserve_password?       [yes]
 Values for check_physical_presence are {[ yes ]| no }.
  Set check_physical_presence? [no] no
  Set check_physical_presence? [no]
 Enter 'y[es]' to commit changes: [no] y
Summary: Changed 2 settings.
Preboot>

4.2.1.2 Edit Mode Settings in Preboot Menu

In addition to changing the settings required in the Preboot Menu to enable remote access, you can also change other edit mode settings in the Preboot Menu. For a list of these settings, see TABLE 4-3.


TABLE 4-3 Edit Mode Preboot Menu Command Settings

Setting

Description

baudrate

Sets the baud rate of the serial port. Selections include 9600,19200, 38400, 57600, and 115200.

serial_is_host

If this is set to 0, the serial port connects to the ILOM. If this is set to 1, the serial port connects to the host. For more details, see Restoring ILOM Access to the Serial Console.

bootdelay

The number of seconds the bootstrap process waits for the user to enter xyzzy before booting the SP. Selections include 3, 10, or 30.

bootretry

The number of seconds the Preboot Menu waits for user input before timing out and starting the SP. Set to -1 to disable the time-out.

preferred

Unused

preserve_conf

Setting this to no duplicates the function of the unconfig ilom_conf command, which resets many ILOM configuration settings, but preserves SP network, baudrate, and check_physical_presence the next time the SP is booted.

preserve_users

Setting this to no duplicates the function of the unconfig users command, which resets user information to the default values the next time the SP is booted.

preserve_
password

Setting this to no duplicates the function of the unconfig password command, which resets the root password to the default value the next time the SP is booted.

check_physical_presence

If this is set to Yes, you must press and hold the Locate button to interrupt the SP boot process. If it is set to No, the boot process prompts you to interrupt it. See Edit Preboot Menu for Remote Serial Access for details.


4.2.2 Restoring the Factory Default ILOM Root Password

The ILOM root password grants you access to the ILOM web interface or command-line interface (CLI) on the SP or chassis monitoring module (CMM). If you forget the root password, you can use the Preboot Menu to restore the password to the factory default, changeme.


procedure icon  Restore ILOM Root Password to Factory Default Using the Preboot Menu

1. Access the Preboot Menu as described in Accessing the Preboot Menu.

2. In Preboot Menu, type:

Preboot> unconfig password

Setting ‘preserve_password’ to ‘no’ for the next boot of ILOM.

3. Reset the SP by typing:

Preboot> boot

The Preboot Menu exits and the SP restarts.

After restarting the ILOM SP, the value for the root password (on the ILOM SP and CMM) is set to changeme when the SP is finished booting.

4.2.3 Restoring ILOM Access to the Serial Console

In the event that the serial connection between ILOM and a host becomes unavailable, you can restore access to the serial port connection by reconfiguring the host as the external serial port owner in either the ILOM web interface or CLI, or in the Preboot Menu.

To determine which interface to use when restoring the serial connection between ILOM and a host console, consider the following:

For instructions, see the procedure for “Switching Serial Port Output Between SP and Host Console” in the Oracle Integrated Lights Out Manager (ILOM) 3.0 Supplement for Sun Netra X6270 M2 Server Module, 821-0943.


procedure icon  Restore Access to the Serial Console Using the Preboot Menu

1. Access the Preboot Menu as described in Accessing the Preboot Menu.

2. At the Preboot> prompt, type edit.

The Preboot Menu enters edit mode.

In edit mode, the Preboot Menu displays its selections one-by-one, offering you a chance to change each one.

3. Press Enter to move through the settings until the serial_is_host setting appears.

To change the serial_is_host setting, type 0, and then press Enter.

The Preboot Menu appears with the new value for the serial_is_host setting.

4. Press Enter to display the Preboot Menu selections.

The Preboot Menu settings appear.

5. Press Enter to scroll through the settings until the Preboot Menu asks you to confirm your changes.

Enter ‘y[es]’ to commit changes: [no]

6. Type y to confirm your change and exit the edit session.

The Preboot Menu displays this message

Summary: Changed 1 settings

Preboot>

7. To exit the Preboot Menu, type boot.

4.2.4 Restoring the SP Firmware Image

If ILOM is available, you should always use the ILOM web interface or CLI to restore the firmware image. For instructions about how to restore the firmware image using either the ILOM web interface or CLI, see the Oracle Integrated Lights Out Manager (ILOM) 3.0 Documentation Collection. If ILOM is unavailable, you can use the Preboot Menu to restore the SP firmware image.



Note - To restore the SP firmware image using the Preboot Menu, you must be an authorized Service technician to perform the procedure.


To use the Preboot Menu to restore the SP firmware image on the server, see the following sections:

4.2.4.1 Prerequisites for Restoring SP Firmware Using the Preboot Menu

The following requirements must be met prior to restoring the SP firmware on your server using the Preboot Menu.



Note - Restoring the SP firmware using the Preboot Menu requires a .flash file instead of a.pkg file that is typically used to update the SP using the ILOM interfaces.



procedure icon  Enabling Support in Preboot Menu to Recover the SP

You must enable support in the Preboot Menu to recover the SP firmware image prior to performing the steps in Restore the SP Firmware Image Using the Preboot Menu.

To enable support in the Preboot Menu to recover SP firmware, follow these steps.

1. Prepare the server for service by powering down the server, removing the server module from the system chassis, then removing the server cover.

For instructions, see the following sections:

2. Locate the J602 pins on the server module then place a jumper on pins 2 and 3. See FIGURE 4-1.

FIGURE 4-1 Placing Jumper on Pins


Figure showing the pin locations on the server module motherboard.

3. Replace the top cover, return the server module to the system chassis and power-on the server.

For instructions, see the following sections:

4. Follow the instructions for restoring the SP firmware using the Preboot Menu in Restore the SP Firmware Image Using the Preboot Menu.



Note - The Preboot Menu firmware recovery process must be performed by an authorized Service technician and you must have a valid .flash file to perform the procedure.


5. After you restore the SP image using the Preboot menu, perform the following steps to remove the J602 jumper from the server and to return the server to normal operation.

a. Power off the server module.

See Power Off and Verify Server Is Ready for Removal From Chassis.

b. Remove the server module from the chassis.

See Remove Server Module From Chassis.

c. Remove the top cover from the server.

See Remove Cover From Server.

d. Remove the jumper from J602. See FIGURE 4-1 for the location of the J602 jumper.

e. Replace the top cover.

See Install Cover on Server.

f. Return the server module into the chassis.

See “Installing the Server Module Into System Chassis” in the Sun Netra X6270 M2 Server Module Installation Guide, 821-0937.

g. Power on the server.

See Power On the Server Module.


procedure icon  Restore the SP Firmware Image Using the Preboot Menu

To restore the SP firmware image using the Preboot Menu, follow these steps:

1. Access the Preboot Menu as described in Accessing the Preboot Menu.

2. At the Preboot> prompt, type:



caution icon Caution - The use of the netflashcommand is reserved for use by authorized Service personnel only.


net flash IPaddress path/name.flash

Where:

For example:

Preboot> net flash 10.8.173.25 images/system-rom.flash

After a series of messages, the Preboot Menu prompt appears.

Preboot>

3. At the Preboot> prompt, type the reset command to restart the SP.

For example:

Preboot> reset

The Preboot Menu exits and ILOM restarts.

4.2.5 Preboot Menu Command Summary

TABLE 4-4 identifies the Preboot Menu commands.


TABLE 4-4 Preboot Menu Commands

Command

Description

boot

Boots ILOM. The Preboot Menu exits and ILOM restarts.

Note - This command executes a modified boot sequence that does not offer the choice to select the diagnostic level, or to interrupt the boot sequence and return to the Preboot Menu. To execute the normal boot sequence, use the reset warm command instead.

vers

Displays version information including the hardware type, board revision, ILOM revision, revisions of PBSW and recovery U-Boot. Shows the checksum integrity of the images, and the preference between redundant images.

help

Displays a list of commands and parameters.

show

Displays a list of SP settings.

edit

Starts an interactive dialog that prompts and changes settings one-by-one. See Edit Mode Settings in Preboot Menu for details.

diag

Runs the U-Boot diagnostic tests in manual mode. See the Oracle x86 Servers Diagnostics Guide, 820-6750 for more on U-Boot diagnostic tests.

host

Initiates various activities related to the host.

 

  • clearcmos - Clears CMOS and BIOS passwords.
  • console - Connects SP console to host serial console.

Note - Type Ctrl \ q to quit.

  • show - Shows information about the host state.
  • enable-on - Enables the front-panel Power button, which is usually disabled unless ILOM is running.

Caution - If you start the host when ILOM is off, the BIOS does not send error events, or power messages to the SP. This can cause all server modules to lose power.

  • hard-off - Turns the host off.

net

{ config | dhcp | ping | flash }

  • config - Starts a dialog that enables you to change the ILOM network settings.
  • dhcp - Changes the network addressing from static to DHCP.

Note - You must set ipdiscovery = dhcp using the net config command first.

Type help net command for more details on these commands.

reset

{ warm | cold }. Resets the SP and the host.

  • warm - Resets the SP without affecting a running host.
  • cold - Resets the SP and the host. It has the effect of powering off the server module.

unconfig

{ users | ilom_conf | most | all }

Causes ILOM to erase any configuration information and returns the values to defaults the next time ILOM boots.

  • users - Resets all configured user information.
  • password - Resets the ILOM root password to the default. See Restoring the Factory Default ILOM Root Password for more details.
  • ilom_conf - Resets configuration settings but preserves SP network and baudrate, preferred, and check_physical_presence.
  • most - Resets the SP data storage, but preserves SP network and baud rate, preferred, and check_physical_presence settings.
  • all - Resets all SP data storage and settings.

Booting ILOM restores other defaults.

Note - None of these options erases the dynamic FRU PROMs.