Chapter 4 Diagnostic Enhancements

This chapter describes the following new system administration information:

• "Improved Core File Management (coreadm)"

• "New Console Features for Remote Systems"

• "Improved System Boot and Error Message Format"

For the most up-to-date man pages, use the man command. The Solaris 7 11/99 man pages include new feature information not found in the Solaris 7 Reference Manual Collection.

Improved Core File Management (coreadm)

This functionality is updated in the Solaris 7 8/99 software release.

Enhanced process core file features that include the coreadm command update the Solaris 7 software release. This information supplements information on troubleshooting software problems found in "Troubleshooting Software Problems (Overview)" in the System Administration Guide, Volume II.

This release introduces the coreadm command, which provides flexible core file naming conventions and better core file retention. For example, you can use the coreadm command to configure a system so that all process core files are placed in a single system directory. This means it is easier to track problems by examining the core files in a specific directory whenever a Solaris process or daemon terminates abnormally.

Limitations of the previous Solaris process core dump features are:

• Process core files are placed in their current working directory, and thus all Solaris daemons, which typically chdir to the root (/) directory as part of their initialization, overwrite each other's core files.

• Many system daemons, such as statd, perform setuid operations but do not, for security reasons, produce core files in the event of a problem.

Configurable Core File Paths

Two new configurable core file paths that can be enabled or disabled independently of each other are:

• A per-process core file path, which defaults to core and is enabled by default. If enabled, the per-process core file path causes a core file to be produced when the process terminates abnormally. The per-process path is inherited by a new process from its parent process.

  When generated, a per-process core file is owned by the owner of the process with read/write permissions for the owner. Only the owning user can view this file.

• A global core file path is disabled by default. If enabled, an additional core file with the same content as the per-process core file is produced by using the global core file path.

  When generated, a global core file is owned by the superuser with read/write permissions for the superuser only. Non-privileged users cannot view this file.

When a process terminates abnormally, it produces a core file in the current directory as in previous Solaris releases. But if the global core file path is enabled and set to /corefiles/core, for example, then each process that expires produces two core files: one in the current working directory and one in the /corefiles directory.

By default, the Solaris core paths and core file retention remain the same:

• A setuid process does not produce core files using either the global or per-process path.

• The global core file path is disabled.

• The per-process core file path is enabled.

• The per-process core file path is set to core.

Expanded Core File Names

If a global core file directory is enabled, core files can be distinguished from one another by using the variables described in the following table.

For example, if the global core file path is set to:

/var/core/core.%f.%p

and a sendmail process with PID 12345 terminates abnormally, it produces the following core file:

/var/core/core.sendmail.12345

Setting the Core File Name Pattern

You can set a core file name pattern on a global basis or a per-process basis. You can specify whether you want these settings saved across a system reboot.

For example, the following coreadm command sets the global core file pattern for all processes started by the init process. This pattern will persist across system reboots.

$ coreadm -i /var/core/core.%f.%p

Global core values are stored in the /etc/coreadm.conf file, which means these settings are saved across a system reboot.

This coreadm command sets the per-process core file name pattern for all processes:

$ coreadm -p /var/core/core.%f.%p $$

The $$ symbols represent a placeholder for the process ID of the currently running shell. The per-process core file name pattern is inherited by all child processes.

Once a global or per-process core file name pattern is set, it must be enabled with the coreadm -e command. See the procedures below for more information.

You can set the core file name pattern for all processes run during a user's login session by putting the command in a user's $HOME/.profile or .login file.

Enabling setuid Programs to Produce Core Files

You can use the coreadm command to enable or disable setuid programs to produce core files for all system processes or on a per-process basis by setting the following paths:

• If the global setuid option is enabled, a global core file path allows all setuid programs on a system to produce core files.

• If the per-process setuid option is enabled, a per-process core file path allows specific setuid processes to produce core files.

By default, both flags are disabled. For security reasons, the global core file path must be a full pathname, starting with a leading /. If superuser disables per-process core files, individual users cannot obtain core files.

The setuid core files are owned by the superuser with read/write permissions for the superuser only. Ordinary users cannot access them even if the process that produced the setuid core file was owned by an ordinary user.

See coreadm(1M) for more information.

How to Display the Current Core Dump Configuration

Use the coreadm command without any options to display the current core dump configuration.

$ coreadm
               global core file pattern: /var/core/core.%f.%p
                 init core file pattern: core
                      global core dumps: enabled
                 per-process core dumps: enabled
                global setid core dumps: enabled
           per-process setid core dumps: disabled
               global core dump logging: disabled

How to Set a Core File Name Pattern

1. Determine whether you want to set a per-process or global core file and select one of the following:

   1. Set a per-process core file name pattern.

      $ coreadm -p $HOME/corefiles/%f.%p $$

   2. Set a global core file name pattern.

      Become superuser first.

      # coreadm -g /var/core/core.%f.%p

How to Display a Core File Name Pattern

Use the following coreadm command to inquire about the core file settings of the current process. The $$ symbols represent a placeholder for the process ID of the running shell.

$ coreadm $$
278:    core.%f.%p

Superuser can inquire about any user's core file settings by using coreadm process ID. Ordinary users can only inquire about the core file settings of their own processes.

How to Enable a Per-Process Core File Path

1. Become superuser.

2. Enable a per-process core file path.

   # coreadm -e process

3. Display the current process core file path to verify the configuration.

   $ coreadm $$ 1180: /home/kryten/corefiles/%f.%p

How to Enable a Global Core File Path

1. Become superuser.

2. Enable a global core file path.

   # coreadm -e global -g /var/core/core.%f.%p

3. Display the current process core file path to verify the configuration.

   # coreadm global core file pattern: /var/core/core.%f.%p init core file pattern: core global core dumps: enabled per-process core dumps: enabled global setid core dumps: disabled per-process setid core dumps: disabled global core dump logging: disabled

Troubleshooting Core File Problems

Error Message

NOTICE: 'set allow_setid_core = 1' in /etc/system is obsolete NOTICE: Use the coreadm command instead of 'allow_setid_core'

Cause

You have an obsolete parameter that allows setuid core files in your /etc/system file.

Solution

Remove allow_setid_core=1 from the /etc/system file. Then use the coreadm command to enable global setuid core file paths.

New Console Features for Remote Systems

This feature was new in the Solaris 7 5/99 software release.

The new console features in the Solaris 7 5/99 software release update the Solaris operating environment. This information supplements information on troubleshooting Solaris systems found in "Troubleshooting Software Problems (Overview)" in System Administration Guide, Volume II and in the Solaris Transition Guide.

The following new console features improve your ability to troubleshoot remote systems:

• The consadm command enables you to select a serial device as an auxiliary (or remote) console. Using the consadm command, a system administrator can configure one or more serial ports to display redirected console messages and to host sulogin sessions when the system transitions between run levels. This feature enables you to dial in to a serial port with a modem to monitor console messages and participate in init state transitions. (See sulogin(1M) and the step-by-step procedures below for more information.)

  While you can log in to a system using a port configured as an auxiliary console, it is primarily an output device displaying information that is also displayed on the default console. If boot scripts or other applications read and write to and from the default console, the write output will display on all the auxiliary consoles, but the input will only be read from the default console. Auxiliary consoles cannot be used to provide input to boot scripts. (See "Using the consadm Command During an Interactive Login Session" for using the consadm command during an interactive login session.)

• Console output now consists of kernel and syslog messages written to a new pseudo device, /dev/sysmsg. In addition, rc script startup messages are written to /dev/msglog. Previously, all of these messages were written to /dev/console.

  Scripts that direct console output to /dev/console need to be changed to /dev/msglog if you want to see script messages displayed on the auxiliary consoles. Source code referencing /dev/console should be explicitly modified to use syslog() or strlog() if you want messages to be redirected to an auxiliary device.

• The consadm command runs a daemon to monitor auxiliary console devices. Any display device designated as an auxiliary console that disconnects--hangs up or loses carrier--is removed from the auxiliary console device list and is no longer active. Enabling one or more auxiliary consoles does not disable message display on the default console; messages continue to display on /dev/console.

Using Auxiliary Console Messaging During Run Level Transitions

Keep the following in mind when using auxiliary console messaging during run level transitions:

• Input cannot come from an auxiliary console if user input is expected for an rc script that is run when a system is booting. The input must come from the default console.

• The sulogin program, invoked by init to prompt for the superuser password when transitioning between run levels, has been modified to send the superuser password prompt to each auxiliary device in addition to the default console device.

• When the system is in single-user mode and one or more auxiliary consoles are enabled using the consadm command, a console login session runs on the first device to supply the correct superuser password to the sulogin prompt. When the correct password is received from a console device, sulogin disables input from all other console devices.

• A message is displayed on the default console and the other auxiliary consoles when one of the consoles assumes single-user privileges. This message indicates which device has become the console by accepting a correct superuser password. If there is a loss of carrier on the auxiliary console running the single-user shell, one of two actions may occur:

  • If the auxiliary console represents a system at run level 1, the system proceeds to the default run level.

  • If the auxiliary console represents a system at run level S, the system displays the ENTER RUN LEVEL (0-6, s or S): message on the device where the init s or shutdown command had been entered from the shell. If there isn't any carrier on that device either, you will have to reestablish carrier and enter the correct run level. The init or shutdown command will not redisplay the run-level prompt.

• If you are logged in to a system using a serial port, and an init or shutdown command is issued to transition to another run level, the login session is lost whether this device is the auxiliary console or not. This situation is identical to Solaris releases without auxiliary console capabilities.

• Once a device is selected as an auxiliary console using the consadm command, it remains the auxiliary console until the system is rebooted or the auxiliary console is unselected. However, the consadm command includes an option to set a device as the auxiliary console across system reboots. (See the procedure below for step-by-step instructions.)

Using the consadm Command During an Interactive Login Session

If you want to run an interactive login session by logging in to a system using a terminal that is connected to a serial port, and then using the consadm command to see the console messages from the terminal, note the following behavior.

• If you use the terminal for an interactive login session while the auxiliary console is active, the console messages are sent to the /dev/sysmsg or /dev/msglog devices.

• While you issue commands on the terminal, input goes to your interactive session and not to the default console (/dev/console).

• If you run the init command to change run levels, the remote console software kills your interactive session and runs the sulogin program. At this point, input is accepted only from the terminal and is treated like it is coming from a console device. This allows you to enter your password to the sulogin program as described in "Using Auxiliary Console Messaging During Run Level Transitions".

  Then, if you enter the correct password on the (auxiliary) terminal, the auxiliary console runs an interactive sulogin session, locks out the default console and any competing auxiliary console. This means the terminal essentially functions as the system console.

• From here you can change to run level 3 or go to another run level. If you change run levels, sulogin runs again on all console devices. If you exit or specify that the system should come up to run level 3, then all auxiliary consoles lose their ability to provide input. They revert to being display devices for console messages.

  As the system is coming up, you must provide information to rc scripts on the default console device. After the system comes back up, the login program runs on the serial ports and you can log back in to another interactive session. If you have designated the device to be an auxiliary console, you will continue to get console messages on your terminal, but all input from the terminal goes to your interactive session.

How to Enable an Auxiliary (Remote) Console

The consadm daemon does not start monitoring the port until after you add the auxiliary console with the consadm command. As a security feature, console messages are only redirected until carrier drops, or the auxiliary console device is unselected. This means carrier must be established on the port before you can successfully use the consadm command.

See the man page consadm(1M) for more information on enabling an auxiliary console.

1. Log in to the system as superuser.

2. Enable the auxiliary console.

   # consadm -a devicename

3. Verify that the current connection is the auxiliary console.

   # consadm

Example--Enabling an Auxiliary (Remote) Console

# consadm -a /dev/term/a 
# consadm
/dev/term/a

How to Display a List of Auxiliary Consoles

1. Log in to the system as superuser.

2. Select one of the following steps:

   1. Display the list of auxiliary consoles.

      # consadm /dev/term/a

      or

   2. Display the list of persistent auxiliary consoles.

      # consadm -p /dev/term/b

How to Enable an Auxiliary (Remote) Console Across System Reboots

1. Log in to the system as superuser.

2. Enable the auxiliary console across system reboots.

   # consadm -a -p devicename

   This adds the device to the list of persistent auxiliary consoles.

3. Verify that the device has been added to the list of persistent auxiliary consoles.

   # consadm

Example--Enabling an Auxiliary (Remote) Console Across System Reboots

# consadm -a -p /dev/term/a  
# consadm
/dev/term/a

How to Disable an Auxiliary (Remote) Console

1. Log in to the system as superuser.

2. Select one of the following steps:

   1. Disable the auxiliary console.

      # consadm -d devicename

      or

   2. Disable the auxiliary console and remove it from the list of persistent auxiliary consoles.

      # consadm -p -d devicename

3. Verify that the auxiliary console has been disabled.

   # consadm

Example--Disabling an Auxiliary (Remote) Console

# consadm -d /dev/term/a 
# consadm

Improved System Boot and Error Message Format

This functionality was new in the Solaris 7 3/99 software release.

This supplements information on system boot and error messages found in "Viewing System Messages" in System Administration Guide, Volume II, and referenced in the Solaris Transition Guide.

The Solaris 7 3/99 release improves the system boot and error message format by providing a numeric identifier, module name, and time stamp to messages generated by the syslog logging facility. In addition, messages that were previously lost after a system panic and reboot are now saved.

The new message format can be enabled or disabled by setting the msgid property in the log.conf file. The new message format is not enabled by default. See the man page, log(7D), and the procedure below for more information.

For general information about system error logging, see the System Administration Guide, Volume II.

System Boot and Error Message Format Changes

If msgid is set to 0 in the log.conf file, there are no changes in the message format. If msgid is set to 1, there are two changes in the message format:

1. The text of the message is preceded by a message ID that looks like this:

   [ID msgid facility.priority]

   For example:

   [ID 123456 kern.notice]

   The msgid identifier is described in the man page msgid(1M). The facility and priority identifiers are described in the man page, syslog.conf(4).

2. If the message originated in the kernel, the kernel module name is displayed instead of just 'unix.'

   Previous message format:

   Oct 1 14:07:24 mars unix: alloc: /: file system full

   New message format:

   Oct 1 14:07:24 mars ufs: [ID 845546 kern.notice] alloc: /: file system full

How to Enable System Message IDs

1. Become superuser.

2. Enable system message IDs by adding the following line to the /platform/`uname -i`/kernel/drv/log.conf file, if it exists. Otherwise, add the msgid property to the /kernel/drv/log.conf file.

   msgid=1

3. Save and close the file.

4. Reboot the system by using the following command.

   # init 6

   ---

   Note -

   To enable system message IDs without rebooting the system, use the following adb command:

   # echo log_msgid/W1 | adb -kw

   ---

How to Disable System Message IDs

1. Become superuser.

2. Disable system message IDs by changing the msgid line in the /platform/`uname -i`/kernel/drv/log.conf file to the following, if it exists. Otherwise, change the msgid property to the /kernel/drv/log.conf file.

   msgid=0

3. Save and close the file.

4. Reboot the system by using the following command.

   # init 6

   ---

   Note -

   To disable system message IDs without rebooting the system, use the following adb command:

   # echo log_msgid/W0 | adb -kw

   ---