C H A P T E R 3 - Troubleshooting the File System Manager

C H A P T E R 3

Troubleshooting the File System Manager

This chapter describes how to troubleshoot issues that might arise when using the File System Manger software.

This chapter covers the following topics:

File System Manager Messages

Log and Trace Files

Remote Procedure Call (RPC) Daemon Information

File System Manager Messages

This section shows some of the messages you might see when using the File System Manager software.

Message:

An unrecoverable error occurred during the page display.If the problem persists, please restart the web server.

Action:

Click the HOME button to return to the Server Selection page, which is the default page of the File System Manager application.

If the system cannot display the Server Selection page, go to the web server and enter the following command to restart the web server:

# /usr/sbin/smcwebserver restart

Contact your Sun support representative if the problem persists.

Message:

HTTP 500 Internal server error

Action:

Go to the web server and run the following command to restart the web server:

# /usr/sbin/smcwebserver restart

Contact your Sun support representative if the problem persists.

Message:

The page cannot be displayed.

Action:

Go to the web server and run the following command to restart the web server:

# /usr/sbin/smcwebserver restart

Contact your Sun support representative if the problem persists.

Message:

Starting Sun(TM) Web Console Version 2.2.4.Startup failed. See /var/log/webconsole/console_debug_log for detailed error information.

Action:

Examine the contents of the following file on the web server:

/var/log/webconsole/console_debug_log

If the log says the port(6789) is in use by some other process, type commands shown in CODE EXAMPLE 0-1.

CODE EXAMPLE 0-1 Restarting the Web Console
# `pkill -9 noaccess` # `/usr/sbin/smcwebserver restart`

Contact your Sun support representative if the problem persists.

Message:

Failed to create the filesystemmount_samfs: fopen(mnttab) error: : Too many open files

Action:

The system generates this message if you are trying to create a file system with a large number of LUNs. To remedy this problem, use the following procedure:

1. On the file system server, use the ps(1) and grep(1) commands to find the process ID for the fsmgmtd process.

For example:

# ps -ef | grep fsmgmtd

2. Use the plimit(1) command to increase the descriptors for the process.

For example:

# plimit -n 512 process-id

For process-id, specify the process number.

3. Create the file system.

Log and Trace Files

The File System Manager software automatically enables logging when it is installed, but if you want tracing, you have to enable it manually. To enable tracing for File System Manager, use the instructions in Tracing.

Log rotation is not supported for log or trace files.

TABLE 3-1 lists the files that the File System Manager uses for logging and tracing.

TABLE 3-1 File System Manager Log and Trace Files
Activity	File Location	Created by the User?
File System Manager Logging	`/var/log/webconsole/fsmgr.log`	No
TomCat Web Console Logging	`/var/log/webconsole/console_debug_log`	No
Tracing for File System Manager and native code	`/var/log/webconsole/fsmgr.trace_syslog`	Yes

The following sections describe the log and trace files.

File System Manager Logging

The File System Manager software creates the fsmgr.log log file when the application starts. It records information about operations that the user performs, and whether those operations were successful. Do not delete or modify this file. If you delete or modify this file, logging stops. When the web server restarts, it erases the contents of this file and creates a new fsmgr.log file.

The File System Manager software uses an additional file, /var/webconsole/fsmgr.log.lck, to ensure that only one process at a time writes to the log file. Do not delete or modify this lock file.

Web Server Logging

The Sun Common Console Framework creates the /var/webconsole/console_debug_log file. It includes console-specific information such as environment variable setting that the console uses and a record of users logged in to the console.

If this file becomes too large, you can delete this file. The system creates another instance of this file the next time the web server restarts.

File System Manager Portal Agent Configuration and Log Files

When the File System Manager software is installed, the File System Manager Portal Agent is also installed. This application acts as a information source for the Sun StorEdge Management Portal application. By default, the File System Manager Portal Agent is disabled. It should only be enabled if you are using the Sun StorEdge Management Portal software. The agent uses the same underlying software as the File System Manager. The following files are used for configuring and logging data from the File System Manager Portal Agent.

/etc/opt/SUNWfsmgr/agent/conf.sh - The configuration script that is used when starting the Tomcat process. It defines location of TomCat, Java and other critical components.

/var/opt/SUNWfsmgr/agent/tomcat/logs - Contains the following log files.

catalina.out - This is the general log file. It contains the log message output from both Tomcat and the agent servlet. Any errors will cause log messages to be written to this file.

fsmgr.<date-stamp>.log - This is the application and servlet log file. It contains messages specific to the loading and the running of the agent servlet. It also contains stack trace and fatal error information from the underlying software.

To verify that the agent is running, check the catalina.out log file or use the ps and grep commands to find the agent process.

# /usr/ucb/ps -augxww | grep SUNWfsmgr/agent/tomcat

Tracing

The File System Manager trace file records the following information:

Messages regarding whether operations were successful.

Functions invoked with the application stack. This can be verbose.

Messages that are important to developers for debugging purposes.

Tracing is not enabled by default.

To Enable Tracing for File System Manager and Native Code

The syslog daemon performs detailed tracing for File System Manager and for native code. Use the following procedure to enable detailed tracing for File System Manager and for native code.

1. Use the touch(1) command to create the trace file.

For example:

# touch /var/log/webconsole/fsmgr.trace_syslog

2. Use vi(1) or another editor to add the following line to the /etc/syslog.conf file:

local6.debug   /var/log/webconsole/fsmgr.trace_syslog

Use a tab character to separate the two fields in this line.

3. Type the following command:

# pkill -HUP syslogd

4. (Optional) Enable trace file rotation.

Trace files can become very large. Use logadm(1M) to manage the trace file for File System Manager.

Note - You cannot use the log_rotate.sh(1M) script to manage the File System Manager trace file.

To Enable Tracing or To Adjust the Tracing Level

Use the following command to enable tracing or to adjust the tracing level:

# /opt/SUNWfsmgr/bin/fsmgr trace trace_level

For trace_level, specify one of the values shown in TABLE 3-2.

TABLE 3-2 Arguments for *trace_level*
trace_level	Tracing Requested
`off`	Disables tracing.
`1`	Enables tracing for very important messages only. This includes severe errors that occur within the application.
`2`	Enables tracing for moderately important messages. This includes level 1 messages as well as debugging statements within the application that are useful for developers.
`3`	Enables tracing for all messages. This includes level 1 and level 2 messages as well as entry and exit points of functions within the application on the stack.

You can enable and disable tracing dynamically during run time by using the fsmgr(1M) command.

File System Manager Portal Agent Tracing

The File System Manager Portal Agent acts as a information source for the Sun StorEdge Management Portal application.The agent uses the same tracing mechanisms as the File System Manager and both applications write to the same trace output file. Enabling tracing will enable tracing for both the File System Manager and the agent. Because of this, it can be hard to decipher the tracing output if both the agent and the File System Manager are running simultaneously. If you need to enable tracing, it is best to have either File System Manager or the agent running but not both.

If you enable tracing and do not see any tracing output for the agent, check the access permissions on the trace log file. The agent application runs as root, so you need to verify that root has access to write to the trace log file.

Remote Procedure Call (RPC) Daemon Information

The following procedure can help you obtain troubleshooting information for the RPC daemon, fsmgmtd(1M).

To Determine Whether the RPC Daemon Is Running

Perform the following procedure to verify whether or not the RPC daemon is running and to obtain status information.

1. Log in to the SAM-QFS server.

2. Become superuser.

3. Display status information for the File System Manager daemon (fsmgmtd).

Enter the following command to display the daemon:

# /opt/SUNWsamfs/sbin/fsmadm status

If the daemon is not running, it does not display its status. Enter the following command to start the daemon:

# /opt/SUNWsamfs/sbin/fsmadm config -a

The preceding command starts the daemon and enables it to automatically restart if it dies.