5 ASR General Troubleshooting

This chapter provides a variety of troubleshooting procedures for the ASR software.

The instructions provided are for Solaris. When possible, corresponding Linux instructions are provided. Please see the appropriate Linux documentation for details for general administration commands.

Note:

To enter the ASR prompt (asr>) as root, type asr on the command line. See Installing ASR Manager Software for instructions for setting the PATH environment variable.

The following troubleshooting topics are presented:

• ASR Status

• ASR Diagnostics

• ASR Manager Crash Recovery

• ASR - No Heartbeat

• ASR Assets for Solaris 11 Troubleshooting

• Resolve ASR Manager Java Path Location in asr.conf File

• Service Tags Troubleshooting

• SMA Service Troubleshooting (Solaris 10 Only)

• Error Messages and Resolutions

• ASR Auto Update Troubleshooting

• ASR Activation Failed Troubleshooting

• Troubleshooting StorageTek Virtual Storage Manager (VSM) Assets

• Troubleshooting ILOM

• Diagnostics Bundle Collection Fails

5.1 ASR Status

You can review the status of any ASR Asset from the ASR Manager or from My Oracle Support. The following ASR Status troubleshooting topics are presented:

• View Status from the ASR Manager

• View Status from My Oracle Support

• ASR Log Files

• Check the State of ASR Bundles

• Check ASR Manager Status

5.1.1 View Status from the ASR Manager

The status of any ASR Asset can be obtained by running any one of the following command options from the ASR Manager system:

ASR Command	Description
list_asset	Lists all assets associated with this ASR Manager.
list_asset -c	Prints the ASR assets list in .csv format. Run the following command to pipe the output into a file: list_asset -c csv_output.csv
list_asset -i <IP address>	Shows the asset associated with the IP address.
list_asset -h <host name>	Shows the asset associated with the host name.
list_asset -s <subnet IP address>	Lists all assets associated with subnet IP address.

Note:

The list_asset command accepts a comma-delimited list of IP addresses, subnets, or host names.

The results will be similar to the following example:

The data in LAST_HEARTBEAT_DATE column can show either NA or a date/time when the ASR Manager received the last heartbeat from the asset.

A value of NA indicates that the ASR Asset never sent a heartbeat to the ASR Manager.

Note:

ASR assets sending individual, asset-specific heartbeats include:

• Solaris 11 asr-notify.

• ILOM version 3.2.2.0 and later for ASR 5.3 and earlier (ASR 5.4 and later do not have this requirement).

• M-10 XSCF.

For other assets, the ASR Manager sends heartbeats on behalf of the asset.

5.1.2 View Status from My Oracle Support

To view the status of all ASR Assets, log in to My Oracle Support (https://support.oracle.com). In the My Oracle Support Dashboard, click the "Systems..." tab. For more information about the ASR Status value, see Figure 4-1, "ASR Asset Status Transition".

In the Settings pane on the left of the window, select Assets (located under the Administrative submenu). A complete list of all ASR Assets is displayed. See the ASR Status column for the status of all ASR assets. Select an asset to view details about the asset, as shown in Figure 5-1:

Figure 5-1 Manage ASR Assets in My Oracle Support (MOS)

5.1.3 ASR Log Files

When you are troubleshooting ASR, you can change the level of information displayed in the logs, and increase or decrease the number of logs that are saved before being overwritten. The logs are written to the asr.log files. Log files are located on the ASR Manager system at /var/opt/asrmanager/log

Log File	Description
asr-http.log	Messages processed by the ASR Manager HTTP receiver.
asr-snmp.log	Activity regarding SNMP traps processing.
asr.log	Error messages and activity regarding the ASR Manager.
auditlog	Audit logs. See ASR Audit Logging.
autoupdate.log	Status updates for the ASR Auto Update feature.
file-upload.log	Activity regarding file uploads processing.
remote-request.log	Activity regarding remote request processing.
service-request.log	Oracle service request numbers created by ASR.
trap-accepted.log	Fault events accepted by the ASR Manager.
trap-rejected.log	Fault events rejected by the ASR Manager.

There are four levels of logs:

1. Debug: Displays the highest level of information. It contains fine, informational, warnings and severe messages.

2. Trace: Displays a more verbose logging than Debug.

3. Info: Displays not only informational data, but also both warnings and severe messages. This is the default setting.

4. Warn: Displays warnings and severe messages.

5. Error: Displays the least amount of information; severe messages only.

The default number of logs collected and saved is 5. Once that number is reached, ASR begins overwriting the oldest file. You have the option to change the number of logs collected and saved. If you are gathering as much information as possible in a short time, you might want to limit the number of logs saved to accommodate the larger files.

5.1.3.1 Set Log Level

Follow the procedure below to set logging levels:

1. Open a terminal window and log in as root on the ASR Manager system.

2. To view the current level of information being gathered, run:

   asr> get_loglevel
   

3. To change the logging level, run:

   asr> set_loglevel [level]
   

   The choices for level are: trace, debug, info, warn, or error.

5.1.3.2 Set Log File Counts

Follow the procedure below to set log file counts:

1. Open a terminal window and log in as root on the ASR Manager system.

2. To view the current number of logs being saved, enter the following command:

   asr> get_logfilecount
   

3. To change the number of logs being saved, enter the following command:

   asr> set_logfilecount [number]
   

5.1.4 Check the State of ASR Bundles

For diagnostic purposes, it may be necessary to check the state of various application bundles installed on the ASR Manager system using the following procedure.

1. Open a terminal window and log in as root to the ASR Manager.

2. Enter the following command:

   asr> lb
   
   START LEVEL 1
      ID|State |Level|Name
       0|Active | 0|System Bundle (4.4.0)
       1|Active | 1|Apache Felix Bundle Repository (1.6.6)
       2|Active | 1|Apache Felix Gogo Command (0.12.0)
       3|Active | 1|Apache Felix Gogo Runtime (0.10.0)
       4|Active | 1|Apache Felix Gogo Shell (0.10.0)
       5|Active | 1|Oracle ASR Transport (5.0.0)
       6|Active | 1|Oracle ASR Database (5.0.0)
       7|Active | 1|Oracle ASR Container (5.0.0)
       8|Active | 1|Oracle ASR ServiceTags (5.0.0)
       9|Active | 1|Oracle ASR Activation (5.0.0)
      10|Active | 1|Oracle ASR SNMP Receiver (5.0.0)
      11|Active | 1|Oracle ASR HTTP Receiver (5.0.0)
      12|Active | 1|Oracle ASR Storage (5.0.0)
      13|Active | 1|Oracle ASR Diagnostics (5.0.0)
      14|Active | 1|Oracle ASR Autoupdate (5.0.0)
      15|Active | 1|Oracle ASR TimerTask Scheduler (5.0.0)
   

3. If any of these bundles are not in an ACTIVE state, enter the following commands:

   asr> stop
   asr> start
   

4. Repeat steps 1 to 3.

5. To ensure everything is working properly, run the following commands:

   asr> test_connection
   asr> send_test
   

5.1.5 Check ASR Manager Status

For diagnostic purposes, it may be necessary to check the status of processes running on the ASR Manager system. For any failures, refer to Error Messages and Resolutions.

To verify the ASR Manager status, run the following script:

/opt/asrmanager/util/check_asr_status.sh

Output of a successful status check should look like this:

Checking ASR Manager status ..
PASS: ASR Manager bundles state is active.
PASS: ASR Manager SNMP listener is running (SNMP port 162).
PASS: ASR Manager database connectivity is working.
PASS: ASR Manager Registration SSO user name is set correctly.
PASS: ASR Manager Oracle transport connectivity is working.
PASS: ASR Manager Oracle transport endpoint is set correctly.
PASS: ASR Manager OSGI port is accessible.
PASS: ASR Manager process is running.

5.2 ASR Diagnostics

To assist with diagnosing issues with ASR Manager installation, configuration, and operation, ASR provides a variety of methods to collect and send the necessary details for resolving any ASR Manager issues. The following topics are provided in support of ASR diagnostics:

• ASR Diagnostic File

• ASR Remote Diagnostics

• Configure the ASR Diagnostic Utility

• ASR Diagnostic Error Messages

5.2.1 ASR Diagnostic File

ASR provides the ability to generate a diagnostic file that can be analyzed by Oracle Support as part of a Service Request, as needed. To generate and send an ASR diagnostic file for analysis with Oracle Support:

1. Create a Service Request in My Oracle Support.

   Note:

   If a valid SR number is not provided, then the upload to Oracle will fail.

2. Run the following command from the ASR Manager:

   asr> send_diag -sr <SR number>
   

   Where the -sr <SR number> is the newly created Service Request number.

   For example:

   asr> send_diag -sr 3-12345678
   
   This command will collect the diagostics file from ASR Manager and upload to Oracle ASR Infrastructure. Do you want to proceed with collect the diagostics bundle? [y/n]: y
   

3. Verify the diagnostic file has been successfully attached to the Service Request. Log in to My Oracle Support and view the Service Request you created earlier. The request should be updated with a new attachment.

4. (Optional) Check the status of the ASR diagnostic file:

   asr> show_log_collection_status
   

   This command displays the ASR diagnostics file's collection status for all collection attempts, either from the ASR command line or from the ASR portal. The collection status is displayed in ascending order.

   Output will look like this:

   asr show_log_collection_status
   
   Diagnostics File Upload Status
   ==========================
   File Name: /var/opt/asrmanager/messages/supportfile/asr-diag-bundle-98F02E0452CBB9F796123917E96CEA10-140915180001.zip
   File Upload Time Stamp: 2014-09-15 18:01:16.713
   Asset Serial: Not Activated
   Service Request Number: 3-123355
   File Uploaded from Client: ASR Manager
   Client Site ID: <client site ID>
   File Upload Status Message: User [e-mail] is not entitled to upload the log files to Oracle ASR Infrastructure. Failure reason: PUT https://host.mycompany.com/upload/issue/3-123355/asr-diag-bundle-98F02E0452CBB9F796123917E96CEA10-140915180001.zip returned a response status of 403 Forbidden
   File Upload Type: Log Collection via Manual Request
   File Upload Requested By: Manual Request from ASR Commandline
   File Type: ASR Manager Diagnostics
   ==========================
   

You can also create a ASR diagnostic file at any time. From the ASR Manager, run the following command and follow the command-line instructions:

/opt/asrmanager/util/diag/asrDiagUtil.sh

Note:

You can specify where the file is to be located. See Configure the ASR Diagnostic Utility for more information. By default, this file is stored in the following directory:

/var/opt/asrmanager/messages/supportfile

5.2.2 ASR Remote Diagnostics

Oracle Support can remotely request diagnostic files that can be analyzed as part of a Service Request, as needed. This feature is enabled by default.

To disable ASR Manager remote diagnostics, run the following command:

asr> disable_remote_request

To enable ASR Manager remote diagnostics, run the following command:

asr> enable_remote_request

5.2.3 Configure the ASR Diagnostic Utility

The diag-config.properties file consists a list of properties for specifying location of the configuration and log directories. It also contains "toggle switches" for enabling and disabling a particular data set to be collected:

• com.sun.svc.asr.util.diag.home.directory – The property for specifying where the diagnostic data .zip bundle will be generated. Default is current directory where the ASR Diagnostic Utility is located.

• com.sun.svc.asr.util.diag.zip.file.prefix – The property for configuring the diagnostic data .zip file's name.

• com.sun.svc.asr.util.diag.zip.recursive property – The property for enabling traversing into subdirectories of any configuration or log directories.

5.2.4 ASR Diagnostic Error Messages

Error Message	Resolution
ASR Manager does not have the Minimum Java version required for the Diagnostics file upload to Oracle ASR Infrastructure. Existing Java Version: 1.6.0_26, Minimum required version: 1.6.0_43	Upgrade the Java version to 1.6.0_43 or above (see Verifying Java Requirements for details). Then point ASR Manager to use this latest Java version. Open the /var/opt/asrmanager/configuration/asr.conf file and edit the java.exec= property to point valid Java path. For example: java.exec=/usr/java/bin/java Save and close the file, then restart the ASR Manager to have the updates take effect: For Solaris: svcadm restart asrm For Linux: service asrm restart
Please enter a valid service request number.	The Service Request (SR) number format should be valid. A valid format is <single digit><-><multiple digits> (for example: 3-1234566). Check the SR number you created and run the send_diag command again with the valid SR number.
Log collection was requested with an invalid SR Number. Cannot upload the logs to Oracle ASR Infrastructure.	The contact registered for the ASR Manager is not authorized to upload diagnostics files to My Oracle Support for this SR. Log in to My Oracle Support to verify the upload permissions.
ClassCastException while uploading file to Oracle ASR Infrastructure. A restart of the ASR Manager is required.	Restart ASR Manager to resolve the issue. For Solaris: svcadm restart asrm For Linux: service asrm restart

5.3 ASR Manager Crash Recovery

In cases where an ASR Manager experiences a critical failure, you can set up a new ASR Manager and reconfigure ASR Assets to report to the new host. The following steps describe a sample scenario:

1. An ASR Manager is set up (e.g., host name: ASRHOST01, IP address: 10.10.10.1) and configured on the network. This ASR host is registered and activated to itself.

2. All ASR assets are configured to report failures to the ASR Manager host (ASRHOST01), and all ASR assets are activated on the host.

3. A critical failure occurs in the cabinet of ASRHOST01 (for example: a fire destroys the system and its data). The assets need to be attached to a different ASR Manager host (e.g., host name: ASRHOST02).

4. A new ASR Manager is set up (e.g., host name: ASRHOST02, IP address: 10.10.10.2) and configured on the network. The new ASR host is registered and activated to itself.

5. All ASR assets are now re-configured to report failures to the new ASR Manager host ASRHOST02, and the trap destination is changed to report failures to ASRHOST02.

6. All ASR assets are now activated on ASRHOST02

Note:

In order to reduce the additional work with moving the ASR Manager to a different location (e.g., from ASRHOST1 to ASRHOST2), you can create an ASR backup on another host or on the existing host. Creating a backup is crucial when recovering from a crash (see "ASR Backup and Restore" for a details on creating an ASR backup).

5.4 ASR - No Heartbeat

Heartbeat is configured to run once every day via an internal timer thread. If there is no response after approximately 48 hours, the unit will be marked as a 'Heartbeat Failure' unit.

You can check to see if any ASR Manager or ASR Asset are in Heartbeat Failure by reviewing the ASR status in My Oracle Support.

If you feel that ASR Manager is configured correctly, then you can troubleshoot your ASR Manager hardware to resolve the problem. See MOS knowledge article 1346328.1 for the instructions to your particular hardware:

https://support.oracle.com/rs?type=doc&id=1346328.1

See the "Heartbeat Failure Notification E-mail Examples" in Auto Service Request (ASR) Email Examples (Doc ID 1963725.1) available in My Oracle Support (https://support.oracle.com):

https://support.oracle.com/rs?type=doc&id=1963725.1

5.5 ASR Assets for Solaris 11 Troubleshooting

In cases where you are having issues with configuring ASR on Solaris 11 assets using the asradm command, then review the status of the following asr-notify SMF service:

svcs asr-notify

Output should look like this:

STATE        STIME      FMRI
online       13:00:31   svc:/system/fm/asr-notify:default

Note:

If the asr-notify service status is in maintenance mode, then clear the maintenance mode:

svcadm clear asr-notify

re-register the Solaris 11 asset with ASR manager

5.6 Resolve ASR Manager Java Path Location in asr.conf File

If you have an incorrect or old version of Java installed, the ASR Manager will not start. The command to start ASR Manager will report the following message (see Start ASR Manager for Solaris and Linux command samples):

*************************************************************************
Warning! An old Java version ( 1.5 ) was detected (tried 
'/usr/jdk/jdk1.5.0_16/bin/java').
Oracle Automated Service Manager requires a Java version of 1.6 or higher
to run correctly.

You can set 'java.exec' property in file
/var/opt/asrmanager/configuration/asr.conf
to point to JAVA 1.7 or later

Java can be downloaded from http://www.java.com
*************************************************************************
 

1. Check the Java version you have installed. From the ASR Manager, run:

   java -version
   

   See Verifying Java Requirements for details of the Java version requirements for ASR. ASR requires Java 7 (1.7.0_13) or later or Oracle Java 8 (1.8.0_25 or later).

2. Get the current Java path location. From the ASR Manager, run:

   cat /var/opt/asrmanager/configuration/asr.conf | grep ^java.exec
   

   The output would look like this:

   java.exec=/usr/bin/java
   

3. Make a backup of the asr.conf file. From the ASR Manager, run:

   cp /var/opt/asrmanager/configuration/asr.conf /var/opt/asrmanager/configuration/asr.conf_<current-timestamp>
   

4. Edit the java.exec property in the asr.conf file to point to the value of the java.exec output from Step 2, which should be for Java 7:

   /usr/jdk/latest/bin/java
   

5. Stop and start ASR Manager. From the ASR Manager, run:

   • For Oracle Solaris:

     svcadm restart asrm
     

   • For Linux:

     service asrm restart
     

5.7 Service Tags Troubleshooting

This section provides a variety of steps to check on the state of the Service Tags that must installed on most ASR systems. If issues arise during the installation and operation of ASR, Service Tags may be part of the issue.

The following Service Tags troubleshooting areas are presented:

• Check the Service Tags

• Check the Service Tags Version

• Check Service Tags Probe

• Check Service Tags Listener

• Unable to Contact Service Tags on Asset

• Unknown or Empty Service Tags on Asset

• Cannot Retrieve the ASR Manager IP Address

• Services are Disabled: stdiscover or stlisten

5.7.1 Check the Service Tags

1. Open a browser window to the system you wish to check using the following command. Be sure to include the / (slash) after agent.

   http://asr_system_hostname:6481/stv1/agent/

2. A response similar to the following will be displayed:

   <st1:response>
   <agent>
   <agent_urn><agent urn number></agent_urn>
   <agent_version>1.1.4</agent_version>
   <registry_version>1.1.4</registry_version>
   <system_info>
   <system>SunOS</system>
   <host><your host name></host>
   <release>5.10</release>
   <architecture>sparc</architecture>
   <platform>SUNW,Sun-Fire-V215::Generic_137111-06</platform>
   <manufacturer>Sun Microsystems, Inc.</manufacturer>
   <cpu_manufacturer>Sun Microsystems, Inc.</cpu_manufacturer>
   <serial_number>0707FL2015</serial_number>
   <hostid><host ID number></hostid>
   </system_info>
   </agent>
   </st1:response>
   

3. If you do not get a response from the Service Tags agent, consult the Service Tags man pages:

   man in.stlisten
   man stclient
   

5.7.2 Check the Service Tags Version

Follow the procedure below to check the Service Tags version:

1. Open a terminal window and log in as root to the ASR system you wish to check.

2. Run the following command to get the Service Tags version:

   stclient -v

ASR requires Service Tags version 1.1.4 or later.

5.7.3 Check Service Tags Probe

Follow the procedure below to determine that the Service Tag discovery probe is running:

1. Open a terminal window and log in as root to the ASR system you wish to check.

2. To determine that the Service Tag discovery probe is running, run the following command:

   svcs -l svc:/network/stdiscover

3. If the probe is running correctly, the following information is displayed:

   fmri svc:/network/stdiscover:default
   name Service Tag discovery probe
   enabled true
   state online
   next_state none
   state_time Wed Sep 03 21:07:28 2008
   restarter svc:/network/inetd:default
   

5.7.4 Check Service Tags Listener

Follow the procedure below to determine that the Service Tags Listener is running:

1. Open a terminal window and log in as root to the ASR system you wish to check.

2. To determine if the Service Tags listener is running, run the following command:

   svcs -l svc:/network/stlisten

3. If the listener is running correctly, the following information is displayed:

   fmri svc:/network/stlisten:default
   name Service Tag Discovery Listener
   enabled true
   state online
   next_state none
   state_time Wed Sep 03 21:07:28 2008
   restarter svc:/network/inetd:default
   xibreXR_US root@s4u-v215c-abc12
   

5.7.5 Unable to Contact Service Tags on Asset

This message indicates that the activation failed during Service Tags discovery. The issue can be either Service Tags is not installed on the ASR Asset or is installed but not running. Also the issue can be network connectivity between ASR Manager and the ASR Asset. Complete the following checks:

1. Check if Service Tags is installed and running on an ASR Asset. Run:

   stclient -x
   

   If you cannot run this command, either Service Tags is not installed or not online.

2. Check if the Service Tags services are installed and online using the following command:

   svcs | grep reg
   

3. The results should be similar to the following example:

   online Aug_23 svc:/application/stosreg:default
   online Aug_23 svc:/application/sthwreg:default
   

4. If you cannot find these services, it means Service Tags is not installed on the ASR asset.

5. If the Service Tags services are online, check if psncollector is online. Run:

   svcs | grep psncollector
   

6. The results should be similar to the following example:

   online Sep_09 svc:/application/psncollector:default
   

7. Make sure that there are no TCP Wrappers installed on the ASR asset to prevent any service tags discovery issues. Run the following command from the ASR Manager system:

   wget http://[assetHostNameOrIPaddress]:6481/stv1/agent/
   

8. If there are TCP wrappers installed on the ASR asset, edit /etc/hosts.allow on the asset by adding:

   in.stlisten:[ASR Manager host name]
   

5.7.6 Unknown or Empty Service Tags on Asset

1. View the ASR Asset's serial number using the following URL:

   http://[AgentipAddress]:6481/stv1/agent/
   

2. If product name is empty or "unknown," then check if the Hardware Service Tags are installed and online. Run:

   svcs | grep sthwreg
   

   The results should look like this:

   online Aug_23 svc:/application/sthwreg:default
   

3. If the serial number is incorrect, contact Oracle Support to resolve the problem.

5.7.7 Cannot Retrieve the ASR Manager IP Address

This error message indicates that the ASR Asset activation failed because the Oracle ASR Manager IP address could not be retrieved. The final step for activating an ASR Asset includes this command:

asr> activate_asset -i [host IP address]

When activation fails, the following error message displays:

Cannot retrieve the SASM IP address, please add the SASM IP address to /etc/hosts

You must edit the /etc/hosts file to update the localhost entry. For example, as root, change an entry that looks like this:

127.0.0.1    hostname123.com hostname123 localhost.localdomain localhost

to this:

127.0.0.1    localhost.localdomain localhost

5.7.8 Services are Disabled: stdiscover or stlisten

Service tag processes (stlisten and stdiscover) must be online in order to activate assets successfully.

1. Check to determine if the stdiscover or stlisten services are disabled. Run the following command:

   svcs stlisten stdiscover

   If the services have been disabled, the output would look like this:

   STATE       STIME         FMRI
   disabled    12:20:14      svc:/network/stdiscover:default
   disabled    12:20:14      svc:/network/stlisten:default
   

2. To enable the stdiscover and stlisten services, run the following command:

   svcadm enable stlisten stdiscover

3. Verify the services are online:

   svcs stlisten stdiscover

   Once the services have been enabled, the output would look like this:

   STATE       STIME         FMRI
   enabled     12:20:14      svc:/network/stdiscover:default
   enabled     12:20:14      svc:/network/stlisten:default
   

5.8 SMA Service Troubleshooting (Solaris 10 Only)

The SMA service needs to be online in order to support Solaris FMA enrichment data properly. Prior to configuring FMA, complete the following steps:

1. To check that the state of the SMA service is online, run:

   svcs sma

2. If SMA is online, the state should indicate online, as in the following example:

   STATE      STIME          FMRI
   online     15:40:31       svc:/application/management/sma:default
   

3. If SMA is not online, run the following command to enable it:

   svcadm enable sma

4. Repeat these steps to confirm SMA is online.

5.9 Error Messages and Resolutions

Error Message	Resolution
WARNING: Unable to retrieve fault details. For additional information and some insights into how to correct, please see the ASR Installation and Operations Guide - located at www.oracle.com/asr. See the ASR General Troubleshooting Section.	Verify that the asset has the right Solaris minimum required version and patch level as per the ASR qualified systems web page (see http://www.oracle.com/asr for more information). Review the community string properties on the asset. ASR Manager requires public as the value of the community string in order to retrieve FMA enrichment and additional fault details. (See Enabling M-Series XSCF Telemetry for more details) Review the FMA trap destination configuration file, and restart sma and fmd SMF services.
WARNING: This trap is rejected because the asset is disabled	Enable the ASR Asset using one of the following commands: asr> enable_asset -i <ip> (where ip is the IP address of the ASR asset) or asr> enable_asset -h <host> (where host is the host name of the ASR asset)
WARNING: this trap is rejected because the asset is not found	Enable the ASR Asset using one of the following commands: asr> activate_asset -i [ip] (where ip is the IP address of the ASR asset) or asr> activate_asset -h [host] (where host is the host name of the ASR asset)
Checking connection to /v1/ _register failed!	Run the asr> register command again. This time, enter 1 or the full URL: https://transport.oracle.com See Registering the ASR Manager.
Failure to Register Errors	The asr.log has more detailed information and a Java stacktrace on what failed during registration. When a failure error is encountered, additional details can be found in: /var/opt/asrmanager/log/asr.log
No Such Host Exception	This error indicates that the host running ASR Manager cannot resolve the IP address for the Data Transport Service server. Refer to Test Connectivity from the ASR Manager to Oracle to troubleshoot and resolve the problem.
Not Authorized. The My Oracle Support account provided could not be verified by the transport server	This error indicates that the communication between transport server and Oracle is down or busy. This can also indicate that the queue set-up is wrong or that the user does not have permissions to the queue.
Socket Exception: Malformed reply from SOCKS server	This error indicates that the socks server is not able to route to the transport server endpoint. Action: Add the correct http proxy information or socks settings. Refer to Configure ASR to Send HTTPS Traffic Through a Proxy Server to correct the information.
Activation failures: This asset cannot be activated. Service Tags on asset abc reports: Product Name: unknown (Invalid Product Name) Serial Number: TEST 123 (Invalid Serial Number)	Valid serial numbers contain letters, digits, period, colons, hyphens, underscores. See Activation Failed for Asset <asset name> Due to Data Error (Solaris 10 Only) for details to correct this issue.
FAIL: Missing Registration SSO username. FAIL: ASR Manager Oracle Transport end point is incorrectly set. FAIL: ASR Manager Oracle Transport connectivity is not working	Refer to Registering the ASR Manager. To verify the ASR Manager status, run the following script: /opt/asrmanager/util/check_asr_status.sh
FAIL: Multiple ASR Manager processes are running.	Check the ASR Manager processes: ps -eaf | grep "gosh.args=-sc" | grep java Output should look like this: root 3898 1 0 14:30:55 ? 0:26 /usr/bin/../java/bin/java -Dgosh.args=-sc telnetd -p6666 start -Xms512m -Xmx153 If ASR Manager is running, kill the processes with the following command: kill -9 [Process_ID] Restart ASR Manager: For Solaris: svcadm restart asrm For Linux: service asrm restart To verify the ASR Manager status, run the following script: /opt/asrmanager/util/check_asr_status.sh
ASR Manager HTTP receiver is not running (HTTP port <http_port>)	See Enabling HTTP Receiver for ASR Manager Relay, Solaris 11, and SDP2 To verify the ASR Manager status, run the following script: /opt/asrmanager/util/check_asr_status.sh

5.9.1 "SNMP GET failed" Error Message

The ASR Manager uses the SNMP GET protocol to query ASR assets for additional fault information (as shown in Figure 5-2).

Figure 5-2 ASR Manager SNMP GET

This is limited to the following products and fault telemetry sources:

• M-Series servers (M3000, M4000, M5000, M8000, M9000), XSCF service processor.

• Solaris 10 on ASR-qualified Oracle servers that require FMA for ASR.

These products send fault events to the ASR Manager using the SNMP TRAP protocol.

The ASR Manager uses the SNMP GET to retrieve additional fault information (such as, FRU part number, serial number, and slot location) from the product. This important information allows Oracle to streamline the service delivery process.

The ASR Manager test_snmp_get command is used to verify SNMP GET connectivity. For example:

asr> test_snmp_get -i <IP Address>
asr> test_snmp_get -h <host name>

Note:

The test_snmp_get command is not supported with an SNMPv3 configuration.

Failure reasons include:

• Incorrect asset configuration.

• Network configuration on routers and firewalls that prohibit SNMP GET traffic.

• For M8000/M9000, the test_snmp_get command has been run against the standby XSCF. The test_snmp_get command should only be run against the active XSCF.

An SNMP GET error message will be returned as:

SNMP GET failed on: asset Hostname/IP

5.9.1.1 Solaris 10 FMA SNMP GET Troubleshooting

To resolve this error for ASR Assets running Solaris 10 FMA:

1. Log in to the ASR Asset.

2. Verify the fmd status:

   # svcs fmd
   

   Output will look like this:

   STATE STIME FMRI
   online Jan_07 svc:/system/fmd:default
   

3. Verify the sma status:

   # svcs sma
   

   Output will look like this:

   STATE STIME FMRI
   online Jan_07 svc:/application/management/sma:default
   

4. Enable fmd and sma:

   # svcadm enable fmd
   # svcadm enable sma
   

5.9.1.2 M-Series Servers XSCF SNMP GET Troubleshooting

To resolve this error for M-Series servers:

1. Log in to the M-Series XSCF.

2. Verify the following information:

   • SNMP is operational with the agent running, accepting requests on port 161.

   • The Service Processor (SP) and Fault Management (FM) Management Information Base ("MIB") is enabled.

   • The community string is set to public in all lower case.

   To verify this information, run:

   XSCF> showsnmp
   

   The output will look like this:

   Agent Status:       Enabled   <<-- Must be "Enabled"
   Agent Port:         161       <<-- Must be "161"
   System Location:    Unknown
   System Contact:     Unknown
   System Description: Unknown
   
   Trap Hosts:
   
   Hostname      Port  Type  Community String   Username   Auth Protocol
   -----------   ----  ----  ----------------   --------   -------------
   10.11.12.13   162    v1   public             n/a        n/a
   
   SNMP V1/V2c:
   
   Status:            Enabled   <<-- Must be "Enabled"
   Community String:  public    <<-- Must be "public" in lower case
   

3. To enable SNMP:

   XSCF> setsnmp enablev1v2c public
   

   Note:

   The SNMP community string is case sensitive. For example, PUBLIC is not the same as public.

   The default community string used by ASR Manager is public.

5.10 ASR Auto Update Troubleshooting

By default, Oracle ASR will download and install the latest version of the ASR software. ASR Auto Update includes a set of error codes to help diagnose and resolve issues you may encounter. See Appendix C, "ASR Auto Update Error Codes," for details.

5.11 ASR Activation Failed Troubleshooting

As part of the activation process (see Activating ASR Assets for details), Oracle ASR automatically checks to verify that the qualified ASR Asset has been properly configured and that telemetry information can be sent. If an ASR Asset fails this activation process, you will receive e-mail notification, depending on the following causes:

• Activation Denied

• Activation Failed for Asset <asset name> Due to Data Error (Solaris 10 Only)

For a complete list of activation-related e-mail samples, see Auto Service Request (ASR) Email Examples (Doc ID 1963725.1) available in My Oracle Support (https://support.oracle.com):

https://support.oracle.com/rs?type=doc&id=1963725.1

5.11.1 Activation Denied

If you receive an "activation denied" e-mail, then check to ensure that the same asset is not already activated by a different ASR Manager. If so, then you must first deactivate that asset from the previous ASR Manager or deactivate that asset in My Oracle Support before re-activating again from a different ASR Manager.

5.11.2 Activation Failed for Asset <asset name> Due to Data Error (Solaris 10 Only)

This message indicates that the message creation failed because of bad or missing data. For an example of the Activation Failed Bad Serial, see Auto Service Request (ASR) Email Examples (Doc ID 1963725.1) available in My Oracle Support (https://support.oracle.com):

https://support.oracle.com/rs?type=doc&id=1963725.1

Most of the time, this error is the result of an incorrect or incomplete serial number or product name.

To troubleshoot this message, complete the following steps:

1. View the ASR Asset's serial number using the following URL:

   http://[AgentipAddress]:6481/stv1/agent/
   

2. If product name is empty or "unknown," then check if the Hardware Service Tags are installed and online. Run:

   svcs | grep sthwreg
   

   The results should look like this:

   online Aug_23 svc:/application/sthwreg:default
   

3. If the serial number is incorrect, contact Oracle Support to resolve the problem.

5.12 Troubleshooting StorageTek Virtual Storage Manager (VSM) Assets

Activate the VSM_SVA ASR Asset with the following command:

asr> activate_storage -d VSM_SVA -i <IP address>

If there are problems, common troubleshooting solutions include:

1. If the activation failed, the output should look like this:

   Failed to configure VSM_SVA device at <IP address>. Can't proceed with activation.
   Please refer to ASR documentation for troubleshooting steps.
   

   To resolve the problem, ensure the device IP address is accessible from the ASR Manager on port 9877. Run the following command:

   telnet <device IP> 9877
   

2. If the activation failed because the device type is unsupported, the output should look like this:

   Cannot activate device. Unsupported Device Type. Svm-sva
   Supported device Types are: VSM_SVA
   

   For example, you would see this output from the following command:

   asr> activate_storage -d Svm-sva -i <IP address>
   

   To resolve the problem, use the supported device type (-d) VSM_SVA.

3. If the IP address is invalid, then output should look like this:

   Failed to configure VSM-SVA device at <IP address> to send alerts to ASR manager. Can't proceed with activation.
   Please check if  <IP address> belongs to a VSM_SVA asset.
   Ensure the asset <IP address> is accessible from ASR manager on port 9877.
   Please refer to ASR documentation for troubleshooting steps.
   

   To resolve the problem, verify that the setup procedures have been completely followed and implemented. Run the following command if the IP address is accessible on port 9877:

   telnet <device IP> 9877
   

4. If the activation failed because the VSM_SVA serial number could not be determined, then the output should look like this:

   Failed to run "status id" command to obtain serial number of VSM_SVA device at <IP address>. Can't proceed with activation.
   Please refer to ASR documentation for troubleshooting steps.
   

   To resolve the problem, ensure that the VSM_SVA asset configuration is done properly. Manually run the status id command on the asset and ensure serial number is properly configured on the asset.

5. If the activation failed because the VSM_SVA asset configuration to send the alerts to the ASR Manager has failed, then the output should look like this:

   Failed to configure VSM-SVA device at <IP address> to send alerts to ASR manager. Can't proceed with activation.Please refer to ASR documentation for troubleshooting steps to manually configure the VSM_SVA asset.
   

   To resolve the problem, you must configure the asset manually to send alerts to the ASR Manager. Run the following commands on the VSM_SVA device:

   vshell -f "rvsadd <asr manager IP> " bye
   vshell -f "rvsstem /var/opt/SUNWsasm/alerts/VSM_SVA" bye
   

5.13 Troubleshooting ILOM

The following sections provide information about troubleshooting the Integrated Lights Out Manager (ILOM):

• Check the Service Tags on ILOM

5.13.1 Check the Service Tags on ILOM

Follow the procedure below to check the Service Tags on ILOM:

1. Log in to the ILOM service processor CLI.

2. To view the ILOM Service Tags properties, enter:

   show /SP/services/servicetag
   

   Output should look like this:

    /SP/services/servicetag
       Targets:
   
       Properties:
           passphrase = none
           servicetag_urn = Q9525
           state = disabled
   
       Commands:
           cd
           set
           show
   

3. To enable Service Tags, you must enable the state property. Run:

   set /SP/services/servicetag state=enabled
   

5.14 Diagnostics Bundle Collection Fails

Problem: Diagnostics bundle collection on ASR Manager may fail with incorrect Java version or java version incompatibility issue.

Resolution: To resolve this problem:

1. Ensure the JAVA environment variable points to the correct Java path. You can set this variable using a symbolic link.

2. Ensure that the ASR Manager java.exec property is pointing to the correct Java path. This property can be set using the following command:

   /opt/asrmanager/bin/asr set_property java.exec <java path>