C H A P T E R  4

Troubleshooting

This chapter provides information to assist you in resolving problems with the Sun StorageTek Backup Manager. It contains the following sections:


Troubleshooting Agents

If you suspect problems with any agent, follow the procedures in this section to isolate the problem.

If you need assistance with resolving the problem, contact Sun technical support http://www.sun.com/service/contacting.

Reviewing the Agent Events Report


The Agent Events Report contains warning and critical level error messages for each agent configured in the Sun StorageTek Backup Manager environment. The report, by default, displays events for all agents generated during the last 24-hour period. You can also view events for a single agent by selecting the agent from the Show Events for: drop-down menu and clicking Refresh.

Agent Events Report screenshot showing events that occurred in the last 24 hours.

You can also check the Messages.log file for events that occurred prior to the last 24-hour period. Each agent's log file is located at:

/opt/sbm/agents/agent_name/log

where agent_name is the name of the agent.

Determining the Status of an Agent

In addition to your daily routine of monitoring the previous night's backups and current tape library status, you also need to verify the status of the data acquisition agents.

Each agent has two status values associated with it:

1. Go to Admin > Data Acquisition Agents and enter the Admin password.

The Agent Status is displayed as shown in the following example.


Agent Status screenshot showing an example of an agent that is down, or not running.

2. Verify the status of each data acquisition agent.

An agent status of up indicates the agent is running and no problems currently exist with the agent.

An agent status of down or unknown indicates the following:


Agent Status

Description

Down

Agent is not running.

Unknown

Indicates a communication error between the specific device agent and the database.


a. If an agent's status is Down or Unknown, open the Agent Event Report to determine the cause of the error.

The events collected during the last 24-hour period for the selected agent are displayed, as shown in the following example.


Agent Status screenshot showing events collected during the last 24-hour period.

b. Determine the source of the problem from the event description, and take the necessary action.

If you cannot determine the source of the problem from this report, gather the information specified in Gathering Data and contact Sun technical support for assistance.

3. Verify the configuration status of each agent.

If an agent's configuration status is any value other than Configured, take the following action depending on its configuration status:


Configuration Status

Description

Not Configured

Indicates the agent is not configured. Configure the agent as described in Configuring an Agent.

In-Progress

Agent has been installed and configuration is in progress.

Error

Indicates a communication error between the device agent and the database. Gather the information specified in Gathering Data and contact Sun support.


Gathering Data

Before you contact Sun technical support, gather the following information for the agent in question:


Diagnosing Communications Problems

If a problem suggests a communications problem between the SBM agent and the backup master server and libraries, check to be sure you have configured connectivity properly. See Appendix B of the Sun StorageTek Backup Manager Installation Guide for detailed procedures.


Troubleshooting Agent Advanced Settings

The Sun StorageTek Backup Manager has built-in debugging messages to assist Sun support personnel when troubleshooting agent problems. If you experience problems with any of the agents that you cannot resolve, contact Sun technical support for assistance.


Running the Agent as the root User

By default, Sun StorageTek Backup Manager installs and runs the agents using the SBM user account. You can also run the agent as root if necessary.

Connecting the Agent Using SSH

Prerequisite: You have already installed the agents as described in the Sun StorageTek Backup Manager Installation Guide.

1. Log in as root user to the SBM server.

2. Change directory to ~/.ssh.

cd ~/.ssh



Note - If the .ssh directory does not exist, run the SSH key generation command to create the directory and accept all default values. This will create two files in the ~/.ssh directory: id_dsa and id_dsa.pub. The file, id_d.pub,has the shared key in it.


ssh-keygen -t dsa

3. Open another session and log in to backup master server as root.

4. Change to the directory: ~/.ssh.

cd ~/.ssh

If the directory does not exist, create it.

run mkdir ~/.ssh

5. Copy the public key from the SBM server to the backup master server:

scp -p id_dsa.pub backup_masterserver_hostname:~/.ssh/id_dsa.pub.sbm

6. On the backup master server, create the file: authorized_keys2:

cd ~/.ssh

cat id_dsa.pub.sbm >> authorized_keys2

cat authorized_keys2

7. From the SBM server, check the connectivity:

ssh backup_server_hostname

ssh backup_server_IP

ssh backup_server_fully_qualified_domain_name

8. When prompted to add each host to the known hosts file, type y.

Result: This will allow the root user to log in from the SBM server to the backup master server without prompting for a password.

Connecting the Agent Using RSH to a UNIX Backup Master Server as root

1. Open the file ~/.rhosts on the backup master server.

2. Edit the ~/.rhosts file to add an entry for the SBM server:

sbmserver_hostname root

3. From the SBM server, check the connectivity:

rsh backup_server_hostname set

Connecting the Agent Using RSH to a Windows Backup Master Server as root

1. Download SFU 3.5 located at:

http://www.microsoft.com/windowsserversystem/sfu/downloads/default.mspx

2. Set up RSH as described in the following instructions:

http://www.microsoft.com/technet/interopmigration/unix/sfu/sfu35rsh.mspx

3. Create a root account on the Windows master server to enable RSH as root from the SBM server.

Follow the steps in the section Installing and Setting Up RSH for Windows of the Sun StorageTek Backup Manager Installation Guide. However, when you edit the .rhosts file (as described in step 5.c.) add an entry for the SBM server using the root user as follows:

sbmserver_hostname root

Running the Agent as root User

1. Check the status of the agent:

svcs agent name

Example:

svcs netbackupAgent

STATE STIME FMRI

online 10:31:25 svc:/application/sbm/agents/netbackupAgent:default

2. If the agent is online, stop the process:

svcadm disable agent_name

3. Verify that the agent is down.

svcs netbackupAgent

ps -ef|grep netbackupAgent

Example:

svcs netbackupAgent

STATE STIME FMRI

disabled 10:26:36 svc:/application/sbm/agents/netbackupAgent:default

ps -ef|grep netbackupAgent

root 18865 18637 0 10:31:13 pts/3 0:00 grep netbackupAgent

4. Go to the directory where the agent is installed.



Note - By default the agents are installed in the /opt/sbm directory.


5. Open the script file for the agent you want to modify:

/opt/sbm/agents/agent name/etc/agent name

Example:

vi /opt/sbm/agents/netbackupAgent/etc/netbackupAgent

6. Go to this line in the script file:

DMN_OPTS="-w ${DAEMON_WD} -msgdir ${DAEMON_LOG} -ini ${DAEMON_INI}/bb.bco -user ${SBM_USER} -group ${SBM_GRP}"

7. Remove the options for sbm user and group. The line will look like this:

DMN_OPTS="-w ${DAEMON_WD} -msgdir ${DAEMON_LOG} -ini ${DAEMON_INI}/bb.bco"

8. Save and close the file.

9. Start the agent.

svcadm enable agent_name

10. Check the agent process to verify that it is running as root.

ps -ef|grep netbackupAgent

Example:

ps -ef|grep netbackupAgent

root 19121 18637 0 10:55:46 pts/3 0:00 grep netbackupAgent

root 18952 1 0 10:32:58 ? 0:03 /opt/sbm/agents/netbackupAgent/bin/netbackupAgent -w /opt/sbm/agents/netbackupAgent


Starting and Stopping the Application Server

You can manually stop and restart the application server if necessary. The situations in which you might have to restart the server include:

Starting the Application Server

1. Log in as root to the SBM server.

2. Log in as sbmapp user:

su - sbmapp

3. Change to the following directory:

cd /opt/sbm/sbmapp/glassfish/bin

4. To start the application server enter the following command:

./asadmin start-domain domain1

Result: A message informs you that the application server has started. If the application server fails to start, contact Sun technical support.

Stopping the Application Server

1. Log in as root to the SBM server.

2. Log in as sbmapp user:

su - sbmapp

3. Change to the following directory:

cd /opt/sbm/sbmapp/glassfish/bin

4. To stop the application server, enter the following command:

./asadmin stop-domain domain1

Result: A message informs you that the application server has stopped.


Changing the GlassFish Server Port

You can configure the GlassFish server to run on a port other than the default port. The following procedure assumes the Web server is running on the default port 4848.

1. Log in to the admin console of the web server.

http://SBM management server:4848



Note - The default login provided by GlassFish are user name: admin and password: adminadmin.


2. Click Application Server from the left menu tree.

3. Click Stop Instance to stop the GlassFish server.

4. Log in as sbmapp user to the SBM server.

5. Go to the GlassFish server installation root directory located at:

/opt/sbm/sbmapp/glassfish

6. Edit the configuration file.

a. Open the file domain.xml, located in the GlassFish server installation root directory:

glassfish_ install_root_directory/domains/domain1/domains/domain1/config/

b. Search for the word port.

c. Specify a new port number.

d. Review all port settings to make sure that all port values contain non-conflicting numbers.

e. Save and close the domain.xml file.

7. Restart the GlassFish server by entering the following command:

glassfish_ install_root_directory/bin/asadmin start-domain

8. To verify the SBM application is running on the new port, log into the web server using the new HTTP listening port:

http://SBM_management_server:new_listening_port

For more information about the GlassFish server, see Chapter 1 (page 25) of the following document:

https://glassfish.dev.java.net/nonav/javaee5/docs/SJSASEEIG.pdf


Removing SBM User Accounts

Prerequisite: Stop the GlassFish server as described in Step 1 through Step 3 in Changing the GlassFish Server Port.

1. Log in as root to the SBM server.

2. Run the userdel command for each SBM user account.

# userdel -r sbm

# userdel -r sbmdb

# userdel -r sbmapp

Result:

3. Run the groupdel command for each SBM user account.

# groupdel sbm

# groupdel sbmapp

# groupdel sbmdb

Result:


SBM Agent Control Commands


TABLE 4-1 SBM Agent Control Commands

Task

Command

Start (enable) an SBM agent

svcadm enable agent_name

Stop (disable) an SBM agent

svcadm disable agent_name

Find the status of an agent

svcs agent_name

Find the state of an agent and its log file location

svcs -l agent_name


SBM Database (PostGres) Control Commands


TABLE 4-2 SBM Database (P ostGres) Control Commands

Task

SMF Command

Start (enable) the SBM database instance

svcadm enable postgresql:sbm

Stop (disable) the SBM database instance

svcadm disable postgresql:sbm

Find the status of the SBM database instance

svcs postgresql:sbm

Find the state the SBM database instance and its log file location

svcs -l postgresql:sbm