5 ASR General Troubleshooting

This chapter provides a variety of troubleshooting procedures for the ASR software.

The instructions provided are for Solaris. When possible, corresponding Linux instructions are provided. Please see the appropriate Linux documentation for details for general administration commands.

Note:

To enter the ASR prompt (asr>) as root, type asr on the command line. See Installing ASR Manager Software for instructions for setting the PATH environment variable.

The following troubleshooting topics are presented:

5.1 ASR Status

You can review the status of any ASR Asset from the ASR Manager or from My Oracle Support. The following ASR Status troubleshooting topics are presented:

5.1.1 View Status from the ASR Manager

The status of any ASR Asset can be obtained by running any one of the following command options from the ASR Manager system:

ASR Command Description
list_asset
Lists all assets associated with this ASR Manager.
list_asset -i <IP address>
Shows the asset associated with the IP address.
list_asset -h <hostname>
Shows the asset associated with the host name.
list_asset -hb
Displays the last heartbeat date.
list_asset -s <subnet IP address>
Lists all assets associated with subnet IP address.

Note:

The list_asset command accepts a comma-delimited list of IP addresses, subnets, or hostnames.

The results will be similar to the following example:

Surrounding text describes asr_status_example.gif.

The output of the list_asset -hb command would be similar to the following example:

SERIAL_NUMBER   HOST_NAME       LAST_HEARTBEAT_DATE       PRODUCT_NAME
-------------   -------------   -----------------------   --------------
ASRBT7777       host-123        NA                        SUNW,Sun-Fire-V210 SPARC System
ASRTEST1        asr-123         NA                        SUN FIRE X2270 x86/x64 System
ASRTST321777    abcd.name.com   2014-03-07 07:39:55.105   SPARC T5-8

The data in LAST_HEARTBEAT_DATE column can show either NA or a date/time when the ASR Manager received the last heartbeat from the asset.

A value of NA indicates that the ASR Asset never sent a heartbeat to the ASR Manager.

Note:

ASR Assets running Solaris 11 and ILOM version 3.2.2.0 will send individual heartbeats.

5.1.2 View Status from My Oracle Support

To view the status of all ASR Assets, log in to My Oracle Support (https://support.oracle.com). In the My Oracle Support Dashboard, click the "Systems..." tab. For more information about the ASR Status value, see Figure 4-1, "ASR Asset Status Transition".

In the Settings pane on the left of the window, select Assets (located under the Administrative submenu). A complete list of all ASR Assets is displayed. See the ASR Status column for the status of all ASR assets. Select an asset to view details about the asset, as shown in Figure 5-1:

Figure 5-1 Manage ASR Assets in My Oracle Support (MOS)

Surrounding text describes Figure 5-1 .

5.1.3 ASR Log Files

When you are troubleshooting ASR, you can change the level of information displayed in the logs, and increase or decrease the number of logs that are saved before being overwritten. The logs are written to the asr.log files. Log files are located on the ASR Manager system at /var/opt/asrmanager/log

Log File Description
asr-http.log Messages processed by the ASR Manager HTTP receiver.
asr-snmp.log Activity regarding SNMP traps processing.
asr.log Error messages and activity regarding the ASR Manager.
auditlog Audit logs. See ASR Audit Logging.
autoupdate.log Status updates for the ASR Auto Update feature.
file-upload.log Activity regarding file uploads processing.
remote-request.log Activity regarding remote request processing.
service-request.log Oracle service request numbers created by ASR.
trap-accepted.log Fault events accepted by the ASR Manager.
trap-rejected.log Fault events rejected by the ASR Manager.

There are four levels of logs:

  1. Debug: Displays the highest level of information. It contains fine, informational, warnings and severe messages.

  2. Trace: Displays a more verbose logging than Debug.

  3. Info: Displays not only informational data, but also both warnings and severe messages. This is the default setting.

  4. Warn: Displays warnings and severe messages.

  5. Error: Displays the least amount of information; severe messages only.

The default number of logs collected and saved is 5. Once that number is reached, ASR begins overwriting the oldest file. You have the option to change the number of logs collected and saved. If you are gathering as much information as possible in a short time, you might want to limit the number of logs saved to accommodate the larger files.

5.1.3.1 Set Log Level

Follow the procedure below to set logging levels:

  1. Open a terminal window and log in as root on the ASR Manager system.

  2. To view the current level of information being gathered, run:

    asr> get_loglevel
    
  3. To change the logging level, run:

    asr> set_loglevel [level]
    

    The choices for level are: trace, debug, info, warn, or error.

5.1.3.2 Set Log File Counts

Follow the procedure below to set log file counts:

  1. Open a terminal window and log in as root on the ASR Manager system.

  2. To view the current number of logs being saved, enter the following command:

    asr> get_logfilecount
    
  3. To change the number of logs being saved, enter the following command:

    asr> set_logfilecount [number]
    

5.1.4 Check the State of ASR Bundles

For diagnostic purposes, it may be necessary to check the state of various application bundles installed on the ASR Manager system using the following procedure.

  1. Open a terminal window and log in as root to the ASR Manager.

  2. Enter the following command:

    asr> lb
    
    START LEVEL 1
       ID|State |Level|Name
        0|Active | 0|System Bundle (4.4.0)
        1|Active | 1|Apache Felix Bundle Repository (1.6.6)
        2|Active | 1|Apache Felix Gogo Command (0.12.0)
        3|Active | 1|Apache Felix Gogo Runtime (0.10.0)
        4|Active | 1|Apache Felix Gogo Shell (0.10.0)
        5|Active | 1|Oracle ASR Transport (5.0.0)
        6|Active | 1|Oracle ASR Database (5.0.0)
        7|Active | 1|Oracle ASR Container (5.0.0)
        8|Active | 1|Oracle ASR ServiceTags (5.0.0)
        9|Active | 1|Oracle ASR Activation (5.0.0)
       10|Active | 1|Oracle ASR SNMP Receiver (5.0.0)
       11|Active | 1|Oracle ASR HTTP Receiver (5.0.0)
       12|Active | 1|Oracle ASR Storage (5.0.0)
       13|Active | 1|Oracle ASR Diagnostics (5.0.0)
       14|Active | 1|Oracle ASR Autoupdate (5.0.0)
       15|Active | 1|Oracle ASR TimerTask Scheduler (5.0.0)
    
  3. If any of these bundles are not in an ACTIVE state, enter the following commands:

    asr> stop
    asr> start
    
  4. Repeat steps 1 to 3.

  5. To ensure everything is working properly, run the following commands:

    asr> test_connection
    asr> send_test
    

5.1.5 Check ASR Manager Status

For diagnostic purposes, it may be necessary to check the status of processes running on the ASR Manager system. For any failures, refer to Error Messages and Resolutions.

To verify the ASR Manager status, run the following script:

/opt/asrmanager/util/check_asr_status.sh

Output of a successful status check should look like this:

Checking ASR Manager status ..
PASS: ASR Manager bundles state is active.
PASS: ASR Manager SNMP listener is running (SNMP port 162).
PASS: ASR Manager database connectivity is working.
PASS: ASR Manager Registration SSO user name is set correctly.
PASS: ASR Manager Oracle transport connectivity is working.
PASS: ASR Manager Oracle transport endpoint is set correctly.
PASS: ASR Manager OSGI port is accessible.
PASS: ASR Manager process is running.

5.2 ASR Diagnostics

To assist with diagnosing issues with ASR Manager installation, configuration, and operation, ASR provides the ability to generate a diagnostic file that can be analyzed by Oracle Support as part of a Service Request, as needed.

To generate and send an ASR diagnostic file for analysis with Oracle Support:

  1. Create a Service Request in My Oracle Support.

    Note:

    If a valid SR number is not provided, then the upload to Oracle will fail.
  2. Run the following command from the ASR Manager:

    asr> send_diag -sr <SR number>
    

    Where the -sr <SR number> is the newly created Service Request number.

    For example:

    asr> send_diag -sr 3-12345678
    
    This command will collect the diagostics file from ASR Manager and upload to Oracle ASR Infrastructure. Do you want to proceed with collect the diagostics bundle? [y/n]: y
    
  3. Verify the diagnostic file has been successfully attached to the Service Request. Log in to My Oracle Support and view the Service Request you created earlier. The request should be updated with a new attachment.

  4. (Optional) Check the status of the ASR diagnostic file:

    asr show_log_collection_status
    

    This command displays the ASR diagnostics file's collection status for all collection attempts, either from the ASR command line or from the ASR portal. The collection status is displayed in ascending order.

    Output will look like this:

    asr show_log_collection_status
    
    Diagnostics File Upload Status
    ==========================
    File Name: /var/opt/asrmanager/messages/supportfile/asr-diag-bundle-98F02E0452CBB9F796123917E96CEA10-140915180001.zip
    File Upload Time Stamp: 2014-09-15 18:01:16.713
    Asset Serial: Not Activated
    Service Request Number: 3-123355
    File Uploaded from Client: ASR Manager
    Client Site ID: <client site ID>
    File Upload Status Message: User asr-contact@mycompany.com is not entitled to upload the log files to Oracle ASR Infrastructure. Failure reason: PUT https://host.mycompany.com/upload/issue/3-123355/asr-diag-bundle-98F02E0452CBB9F796123917E96CEA10-140915180001.zip returned a response status of 403 Forbidden
    File Upload Type: Log Collection via Manual Request
    File Upload Requested By: Manual Request from ASR Commandline
    File Type: ASR Manager Diagnostics
    ==========================
    

You can also create a ASR diagnostic file at any time. From the ASR Manager, run the following command and follow the command-line instructions:

asr> asrDiagUtil.sh

Note:

You can specify where the file is to be located. See Configure the ASR Diagnostic Utility for more information. By default, this file is stored in the following directory:
/opt/asrmanager/util/diag/

5.2.1 Configure the ASR Diagnostic Utility

The diag-config.properties file consists a list of properties for specifying location of the configuration and log directories. It also contains "toggle switches" for enabling and disabling a particular data set to be collected:

  • com.sun.svc.asr.util.diag.home.directory – The property for specifying where the diagnostic data .zip bundle will be generated. Default is current directory where the ASR Diagnostic Utility is located.

  • com.sun.svc.asr.util.diag.zip.file.prefix – The property for configuring the diagnostic data .zip file's name.

  • com.sun.svc.asr.util.diag.zip.recursive property – The property for enabling traversing into subdirectories of any configuration or log directories.

5.2.2 ASR Diagnostic Error Messages

Error Message Resolution
ASR Manager does not have the Minimum Java version required for the 
Diagnostics file upload to Oracle ASR Infrastructure. 
Existing Java Version: 1.6.0_26, Minimum required version: 1.6.0_43
Upgrade the Java version to 1.6.0_43 or above. Then point ASR Manager to use this latest Java version. Open the /var/opt/asrmanager/configuration/asr.conf file and edit the java.exec= property to point valid Java path.

For example:

java.exec=/usr/java/bin/java

Save and close the file, then restart the ASR Manager to have the updates take effect:

  • For Solaris: svcadm restart asrm

  • For Linux: service asrm restart

Please enter a valid service request number.
The Service Request (SR) number format should be valid. A valid format is <single digit><-><multiple digits> (for example: 3-1234566).

Check the SR number you created and run the send_diag command again with the valid SR number.

Log collection was requested with an invalid SR Number. 
Cannot upload the logs to Oracle ASR Infrastructure.
The contact registered for the ASR Manager is not authorized to upload diagnostics files to My Oracle Support for this SR.

Log in to My Oracle Support to verify the upload permissions.

ClassCastException while uploading file to Oracle ASR Infrastructure.
A restart of the ASR Manager is required.
Restart ASR Manager to resolve the issue.

For Solaris: svcadm restart asrm

For Linux: service asrm restart


5.3 ASR Manager Crash Recovery

In cases where an ASR Manager experiences a critical failure, you can set up a new ASR Manager and reconfigure ASR Assets to report to the new host. The following steps describe a sample scenario:

  1. An ASR Manager is set up (e.g., hostname: ASRHOST01, IP address: 10.10.10.1) and configured on the network. This ASR host is registered and activated to itself.

  2. All ASR assets are configured to report failures to the ASR Manager host (ASRHOST01), and all ASR assets are activated on the host.

  3. A critical failure occurs in the cabinet of ASRHOST01 (for example: a fire destroys the system and its data). The assets need to be attached to a different ASR Manager host (e.g., hostname: ASRHOST02).

  4. A new ASR Manager is set up (e.g., hostname: ASRHOST02, IP address: 10.10.10.2) and configured on the network. The new ASR host is registered and activated to itself.

  5. All ASR assets are now re-configured to report failures to the new ASR Manager host ASRHOST02, and the trap destination is changed to report failures to ASRHOST02.

  6. All ASR assets are now activated on ASRHOST02

Note:

In order to reduce the additional work with moving the ASR Manager to a different location (e.g., from ASRHOST1 to ASRHOST2), you can create an ASR backup on another host or on the existing host. Creating a backup is crucial when recovering from a crash (see "ASR Backup and Restore" for a details on creating an ASR backup).

5.4 ASR - No Heartbeat

Heartbeat is configured to run once every day via an internal timer thread. If there is no response after approximately 48 hours, the unit will be marked as a 'Heartbeat Failure' unit.

You can check to see if any ASR Manager or ASR Asset are in Heartbeat Failure by reviewing the ASR status in My Oracle Support.

If you feel that ASR Manager is configured correctly, then you can troubleshoot your ASR Manager hardware to resolve the problem. See MOS knowledge article 1346328.1 for the instructions to your particular hardware:

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=HOWTO&id=1346328.1">>https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=HOWTO&id=1346328.1

See Appendix A, "Heartbeat Failure Notification E-mail Examples" for an e-mail example you may receive should this problem occur.

5.5 ASR Assets for Solaris 11 Troubleshooting

In cases where you are having issues with configuring ASR on Solaris 11 assets using the asradm command, then review the status of the following asr-notify SMF service:

svcs asr-notify

Output should look like this:

STATE        STIME      FMRI
online       13:00:31   svc:/system/fm/asr-notify:default

Note:

If the asr-notify service status is in maintenance mode, then clear the maintenance mode:
svcadm clear asr-notify

re-register the Solaris 11 asset with ASR manager

5.6 Resolve ASR Manager Java Path Location in asr.conf File

If you have an incorrect or old version of Java installed, the ASR Manager will not start. The command to start ASR Manager will report the following message (see Start ASR Manager for Solaris and Linux command samples):

*************************************************************************
Warning! An old Java version ( 1.5 ) was detected (tried 
'/usr/jdk/jdk1.5.0_16/bin/java').
Oracle Automated Service Manager requires a Java version of 1.6 or higher
to run correctly.

You can set 'java.exec' property in file
/var/opt/asrmanager/configuration/asr.conf
to point to JAVA 1.7 or later

Java can be downloaded from http://www.java.com
*************************************************************************
 
  1. Check the Java version you have installed. From the ASR Manager, run:

    java -version
    

    See Verifying Java Requirements for details of the Java version requirements for ASR. ASR requires Java 7 (1.7.0_13) or later or Oracle Java 8 (1.8.0_25 or later).

  2. Get the current Java path location. From the ASR Manager, run:

    cat /var/opt/asrmanager/configuration/asr.conf | grep ^java.exec
    

    The output would look like this:

    java.exec=/usr/bin/java
    
  3. Make a backup of the asr.conf file. From the ASR Manager, run:

    cp /var/opt/asrmanager/configuration/asr.conf /var/opt/asrmanager/configuration/asr.conf_<current-timestamp>
    
  4. Edit the java.exec property in the asr.conf file to point to the value of the java.exec output from Step 2, which should be for Java 7:

    /usr/jdk/latest/bin/java
    
  5. Stop and start ASR Manager. From the ASR Manager, run:

    • For Oracle Solaris:

      svcadm restart asrm
      
    • For Linux:

      service asrm restart
      

5.7 Service Tools Bundle (STB) Troubleshooting (Solaris 10 Only)

This section provides a variety of steps to check on the state of the Service Tools Bundle (STB) that must installed on most ASR systems. If issues arise during the installation and operation of ASR, STB may be part of the issue.

The following STB troubleshooting areas are presented:

5.7.1 Check the Service Tags

  1. Open a browser window to the system you wish to check using the following command. Be sure to include the / (slash) after agent.

    http://asr_system_hostname:6481/stv1/agent/

  2. A response similar to the following will be displayed:

    <st1:response>
    <agent>
    <agent_urn><agent urn number></agent_urn>
    <agent_version>1.1.4</agent_version>
    <registry_version>1.1.4</registry_version>
    <system_info>
    <system>SunOS</system>
    <host><your host name></host>
    <release>5.10</release>
    <architecture>sparc</architecture>
    <platform>SUNW,Sun-Fire-V215::Generic_137111-06</platform>
    <manufacturer>Sun Microsystems, Inc.</manufacturer>
    <cpu_manufacturer>Sun Microsystems, Inc.</cpu_manufacturer>
    <serial_number>0707FL2015</serial_number>
    <hostid><host ID number></hostid>
    </system_info>
    </agent>
    </st1:response>
    
  3. If you do not get a response from the Service Tags agent, consult the Service Tags man pages:

    man in.stlisten
    man stclient
    

5.7.2 Check the Service Tags Version

Follow the procedure below to check the Service Tags version:

  1. Open a terminal window and log in as root to the ASR system you wish to check.

  2. Run the following command to get the Service Tags version:

    stclient -v

ASR requires Service Tags version 1.1.4 or later.

5.7.3 Check Service Tags Probe

Follow the procedure below to determine that the Service Tag discovery probe is running:

  1. Open a terminal window and log in as root to the ASR system you wish to check.

  2. To determine that the Service Tag discovery probe is running, run the following command:

    svcs -l svc:/network/stdiscover

  3. If the probe is running correctly, the following information is displayed:

    fmri svc:/network/stdiscover:default
    name Service Tag discovery probe
    enabled true
    state online
    next_state none
    state_time Wed Sep 03 21:07:28 2008
    restarter svc:/network/inetd:default
    

5.7.4 Check Service Tags Listener

Follow the procedure below to determine that the Service Tags Listener is running:

  1. Open a terminal window and log in as root to the ASR system you wish to check.

  2. To determine if the Service Tags listener is running, run the following command:

    svcs -l svc:/network/stlisten

  3. If the listener is running correctly, the following information is displayed:

    fmri svc:/network/stlisten:default
    name Service Tag Discovery Listener
    enabled true
    state online
    next_state none
    state_time Wed Sep 03 21:07:28 2008
    restarter svc:/network/inetd:default
    xibreXR_US root@s4u-v215c-abc12
    

5.7.5 Unable to Contact Service Tags on Asset

This message indicates that the activation failed during Service Tags discovery. The issue can be either Service Tags is not installed on the ASR Asset or is installed but not running. Also the issue can be network connectivity between ASR Manager and the ASR Asset. Complete the following checks:

  1. Check if Service Tags is installed and running on an ASR Asset. Run:

    stclient -x
    

    If you cannot run this command, either Service Tags is not installed or not online.

  2. Check if the Service Tags services are installed and online using the following command:

    svcs | grep reg
    
  3. The results should be similar to the following example:

    online Aug_23 svc:/application/stosreg:default
    online Aug_23 svc:/application/sthwreg:default
    
  4. If you cannot find these services, it means Service Tags is not installed on the ASR asset.

  5. If the Service Tags services are online, check if psncollector is online. Run:

    svcs | grep psncollector
    
  6. The results should be similar to the following example:

    online Sep_09 svc:/application/psncollector:default
    
  7. Make sure that there are no TCP Wrappers installed on the ASR asset to prevent any service tags discovery issues. Run the following command from the ASR Manager system:

    wget http://[assetHostNameOrIPaddress]:6481/stv1/agent/
    
  8. If there are TCP wrappers installed on the ASR asset, edit /etc/hosts.allow on the asset by adding:

    in.stlisten:[ASR Manager host name]
    

5.7.6 Unknown or Empty Service Tags on Asset

  1. View the ASR Asset's serial number using the following URL:

    http://[AgentipAddress]:6481/stv1/agent/
    
  2. If product name is empty or "unknown," then check if the Hardware Service Tags are installed and online. Run:

    svcs | grep sthwreg
    

    The results should look like this:

    online Aug_23 svc:/application/sthwreg:default
    
  3. If the serial number is incorrect, contact Oracle Support to resolve the problem.

5.7.7 Cannot Retrieve the ASR Manager IP Address

This error message indicates that the ASR Asset activation failed because the Oracle ASR Manager IP address could not be retrieved. The final step for activating an ASR Asset includes this command:

asr> activate_asset -i [host IP address]

When activation fails, the following error message displays:

Cannot retrieve the SASM IP address, please add the SASM IP address to /etc/hosts

You must edit the /etc/hosts file to update the localhost entry. For example, as root, change an entry that looks like this:

127.0.0.1    hostname123.com hostname123 localhost.localdomain localhost

to this:

127.0.0.1    localhost.localdomain localhost

5.7.8 Services are Disabled: stdiscover or stlisten

Service tag processes (stlisten and stdiscover) must be online in order to activate assets successfully.

  1. Check to determine if the stdiscover or stlisten services are disabled. Run the following command:

    svcs stlisten stdiscover

    If the services have been disabled, the output would look like this:

    STATE       STIME         FMRI
    disabled    12:20:14      svc:/network/stdiscover:default
    disabled    12:20:14      svc:/network/stlisten:default
    
  2. To enable the stdiscover and stlisten services, run the following command:

    svcadm enable stlisten stdiscover

  3. Verify the services are online:

    svcs stlisten stdiscover

    Once the services have been enabled, the output would look like this:

    STATE       STIME         FMRI
    enabled     12:20:14      svc:/network/stdiscover:default
    enabled     12:20:14      svc:/network/stlisten:default
    

5.8 SMA Service Troubleshooting (Solaris 10 Only)

The SMA service needs to be online in order to support Solaris FMA enrichment data properly. Prior to configuring FMA, complete the following steps:

  1. To check that the state of the SMA service is online, run:

    svcs sma

  2. If SMA is online, the state should indicate online, as in the following example:

    STATE      STIME          FMRI
    online     15:40:31       svc:/application/management/sma:default
    
  3. If SMA is not online, run the following command to enable it:

    svcadm enable sma

  4. Repeat these steps to confirm SMA is online.

5.9 Error Messages and Resolutions

Error Message Resolution
WARNING: Unable to retrieve fault details. For additional information and some insights into how to correct, please see the ASR Installation and Operations Guide - located at www.oracle.com/asr. See the ASR General Troubleshooting Section.
  1. Verify that the asset has the right Solaris minimum required version and patch level as per the ASR qualified systems web page (see http://www.oracle.com/asr for more information).
  2. Review the community string properties on the asset. ASR Manager requires public as the value of the community string in order to retrieve FMA enrichment and additional fault details. (See Enabling M-Series XSCF Telemetry for more details)

  3. Review the FMA trap destination configuration file, and restart sma and fmd SMF services.

WARNING: This trap is rejected because the asset is disabled Enable the ASR Asset using one of the following commands:

asr> enable_asset -i <ip>

(where ip is the IP address of the ASR asset)

or

asr> enable_asset -h <host>

(where host is the hostname of the ASR asset)

WARNING: this trap is rejected because the asset is not found Enable the ASR Asset using one of the following commands:

asr> activate_asset -i [ip]

(where ip is the IP address of the ASR asset)

or

asr> activate_asset -h [host]

(where host is the hostname of the ASR asset)

Checking connection to /v1/

_register failed!

Run the asr> register command again. This time, enter 1 or the full URL: https://transport.oracle.com

See Registering the ASR Manager.

Failure to Register Errors The asr.log has more detailed information and a Java stacktrace on what failed during registration. When a failure error is encountered, additional details can be found in:
/var/opt/asrmanager/log/asr.log
No Such Host Exception This error indicates that the host running ASR Manager cannot resolve the IP address for the Data Transport Service server. Refer to Test Connectivity from the ASR Manager to Oracle to troubleshoot and resolve the problem.
Not Authorized. The My Oracle Support account provided could not be verified by the transport server This error indicates that the communication between transport server and Oracle is down or busy. This can also indicate that the queue set-up is wrong or that the user does not have permissions to the queue.
Socket Exception: Malformed reply from SOCKS server This error indicates that the socks server is not able to route to the transport server endpoint.

Action: Add the correct http proxy information or socks settings. Refer to Configure ASR to Send HTTPS Traffic Through a Proxy Server to correct the information.

Activation failures:
This asset cannot be activated.
Service Tags on asset abc reports:
Product Name: unknown (Invalid Product Name)
Serial Number: TEST 123 (Invalid Serial Number)
Valid serial numbers contain letters, digits, period, colons, hyphens, underscores.

See Activation Failed for Asset <asset name> Due to Data Error (Solaris 10 Only) for details to correct this issue.

FAIL: Missing Registration SSO username.
FAIL: ASR Manager Oracle Transport end point is incorrectly set.
FAIL: ASR Manager Oracle Transport connectivity is not working
Refer to Registering the ASR Manager.

To verify the ASR Manager status, run the following script:

/opt/asrmanager/util/check_asr_status.sh
FAIL: Multiple ASR Manager processes are running.
  1. Check the ASR Manager processes:
    ps -eaf | grep "gosh.args=-sc" | grep java
    

    Output should look like this:

    root 3898 1 0 14:30:55 ? 0:26 /usr/bin/../java/bin/java -Dgosh.args=-sc telnetd -p6666 start -Xms512m -Xmx153
    
  2. If ASR Manager is running, kill the processes with the following command:

    kill -9 [Process_ID]
    
  3. Restart ASR Manager:

    For Solaris: svcadm restart asrm

    For Linux: service asrm restart

  4. To verify the ASR Manager status, run the following script:

    /opt/asrmanager/util/check_asr_status.sh
    
ASR Manager HTTP receiver is not running (HTTP port <http_port>)
See Enabling HTTP Receiver for ASR Manager Relay, Solaris 11, and VOP

To verify the ASR Manager status, run the following script:

/opt/asrmanager/util/check_asr_status.sh

5.9.1 "SNMP GET failed" Error Message

The ASR Manager uses the SNMP GET protocol to query ASR assets for additional fault information (as shown in Figure 5-2).

Figure 5-2 ASR Manager SNMP GET

Surrounding text describes Figure 5-2 .

This is limited to the following products and fault telemetry sources:

  • M-Series servers (M3000, M4000, M5000, M8000, M9000), XSCF service processor.

  • Solaris 10 on ASR-qualified Oracle servers that require FMA for ASR.

These products send fault events to the ASR Manager using the SNMP TRAP protocol.

The ASR Manager uses the SNMP GET to retrieve additional fault information (such as, FRU part number, serial number, and slot location) from the product. This important information allows Oracle to streamline the service delivery process. For example:

snmp_get -i <IP Address>
snmp_get -h <Hostname>

The ASR Manager test_snmp_get command is used to verify SNMP GET connectivity. For example:

test_snmp_get -i <IP Address>
test_snmp_get -h <Hostname>

Failure reasons include:

  • Incorrect asset configuration.

  • Network configuration on routers and firewalls that prohibit SNMP GET traffic.

An SNMP GET error message will be returned as:

SNMP GET failed on: asset Hostname/IP

5.9.1.1 Solaris 10 FMA SNMP GET Troubleshooting

To resolve this error for ASR Assets running Solaris 10 FMA:

  1. Log in to the ASR Asset.

  2. Verify the fmd status:

    # svcs fmd
    

    Output will look like this:

    STATE STIME FMRI
    online Jan_07 svc:/system/fmd:default
    
  3. Verify the sma status:

    # svcs sma
    

    Output will look like this:

    STATE STIME FMRI
    online Jan_07 svc:/application/management/sma:default
    
  4. Enable fmd and sma:

    # svcadm enable fmd
    # svcadm enable sma
    

5.9.1.2 M-Series Servers XSCF SNMP GET Troubleshooting

To resolve this error for M-Series servers:

  1. Log in to the M-Series XSCF.

  2. Verify the following information:

    • SNMP is operational with the agent running, accepting requests on port 161.

    • The Service Processor (SP) and Fault Management (FM) Management Information Base ("MIB") is enabled.

    • The community string is set to public in all lower case.

    To verify this information, run:

    XSCF> showsnmp
    

    The output will look like this:

    Agent Status:       Enabled   <<-- Must be "Enabled"
    Agent Port:         161       <<-- Must be "161"
    System Location:    Unknown
    System Contact:     Unknown
    System Description: Unknown
    
    Trap Hosts:
    
    Hostname      Port  Type  Community String   Username   Auth Protocol
    -----------   ----  ----  ----------------   --------   -------------
    10.11.12.13   162    v1   public             n/a        n/a
    
    SNMP V1/V2c:
    
    Status:            Enabled   <<-- Must be "Enabled"
    Community String:  public    <<-- Must be "public" in lower case
    
  3. To enable SNMP:

    XSCF> setsnmp enablev1v2c public
    

    Note:

    The SNMP community string is case sensitive. For example, PUBLIC is not the same as public.

    The default community string used by ASR Manager is public.

5.10 ASR Auto Update Troubleshooting

By default, Oracle ASR will download and install the latest version of the ASR software. The following sections provide potential solutions for problems that may arise:

5.10.1 ASR Auto Update Error Codes

ASR Auto Update includes a set of error codes to help diagnose and resolve issues you may encounter. See Oracle ASR: Auto Update Error Codes (Doc ID 1568407.1) in My Oracle Support:

https://support.oracle.com

5.10.2 SELINUX Environment Variable (Linux Only)

If the SELINUX environment variable is set to "Enforcing," then the ASR Auto Update will not be able to upgrade (remove and install) the ASR software .rpm file.

To resolve this issue, change this variable to "Permissive." Run the setenforce Permissive command to enable ASR Auto Update to remove and install the .rpm file automatically.

5.11 ASR Activation Failed Troubleshooting

As part of the activation process (see Activating ASR Assets for details), Oracle ASR automatically checks to verify that the qualified ASR Asset has been properly configured and that telemetry information can be sent. If an ASR Asset fails this activation process, you will receive e-mail notification, depending on the following causes:

For a complete list of activation-related e-mail samples, including activation failures, see ASR Activation E-mail Examples.

5.11.1 Activation Denied

If you receive an "activation denied" e-mail, (as shown in the Activation Denied e-mail sample), then check to ensure that the same asset is not already activated by a different ASR Manager. If so, then you must first deactivate that asset from the previous ASR Manager or deactivate that asset in My Oracle Support before re-activating again from a different ASR Manager.

5.11.2 Activation Failed for Asset <asset name> Due to Data Error (Solaris 10 Only)

This message indicates that the message creation failed because of bad or missing data. See Activation Failed Bad Serial for a sample of this activation-failure e-mail. Most of the time, this error is the result of an incorrect or incomplete serial number or product name. To troubleshoot this message, complete the following steps:

  1. View the ASR Asset's serial number using the following URL:

    http://[AgentipAddress]:6481/stv1/agent/
    
  2. If product name is empty or "unknown," then check if the Hardware Service Tags are installed and online. Run:

    svcs | grep sthwreg
    

    The results should look like this:

    online Aug_23 svc:/application/sthwreg:default
    
  3. If the serial number is incorrect, contact Oracle Support to resolve the problem.

5.12 Troubleshooting StorageTek Virtual Storage Manager (VSM) Assets

Activate the VSM_SVA ASR Asset with the following command:

asr> activate_storage -d VSM_SVA -i <IP address>

If there are problems, common troubleshooting solutions include:

  1. If the activation failed, the output should look like this:

    Failed to configure VSM_SVA device at <IP address>. Can't proceed with activation.
    Please refer to ASR documentation for troubleshooting steps.
    

    To resolve the problem, ensure the device IP address is accessible from the ASR Manager on port 9877. Run the following command:

    telnet <device IP> 9877
    
  2. If the activation failed because the device type is unsupported, the output should look like this:

    Cannot activate device. Unsupported Device Type. Svm-sva
    Supported device Types are: VSM_SVA
    

    For example, you would see this output from the following command:

    asr> activate_storage -d Svm-sva -i <IP address>
    

    To resolve the problem, use the supported device type (-d) VSM_SVA.

  3. If the IP address is invalid, then output should look like this:

    Failed to configure VSM-SVA device at <IP address> to send alerts to ASR manager. Can't proceed with activation.
    Please check if  <IP address> belongs to a VSM_SVA asset.
    Ensure the asset <IP address> is accessible from ASR manager on port 9877.
    Please refer to ASR documentation for troubleshooting steps.
    

    To resolve the problem, verify that the setup procedures have been completely followed and implemented. Run the following command if the IP address is accessible on port 9877:

    telnet <device IP> 9877
    
  4. If the activation failed because the VSM_SVA serial number could not be determined, then the output should look like this:

    Failed to run "status id" command to obtain serial number of VSM_SVA device at <IP address>. Can't proceed with activation.
    Please refer to ASR documentation for troubleshooting steps.
    

    To resolve the problem, ensure that the VSM_SVA asset configuration is done properly. Manually run the status id command on the asset and ensure serial number is properly configured on the asset.

  5. If the activation failed because the VSM_SVA asset configuration to send the alerts to the ASR Manager has failed, then the output should look like this:

    Failed to configure VSM-SVA device at <IP address> to send alerts to ASR manager. Can't proceed with activation.Please refer to ASR documentation for troubleshooting steps to manually configure the VSM_SVA asset.
    

    To resolve the problem, you must configure the asset manually to send alerts to the ASR Manager. Run the following commands on the VSM_SVA device:

    vshell -f "rvsadd <asr manager IP> " bye
    vshell -f "rvsstem /var/opt/SUNWsasm/alerts/VSM_SVA" bye
    

5.13 Troubleshooting ILOM

The following sections provide information about troubleshooting the Integrated Lights Out Manager (ILOM):

5.13.1 Check the Service Tags on ILOM

Follow the procedure below to check the Service Tags on ILOM:

  1. Log in to the ILOM service processor CLI.

  2. To view the ILOM Service Tags properties, enter:

    show /SP/services/servicetag
    

    Output should look like this:

     /SP/services/servicetag
        Targets:
    
        Properties:
            passphrase = none
            servicetag_urn = Q9525
            state = disabled
    
        Commands:
            cd
            set
            show
    
  3. To enable Service Tags, you must enable the state property. Run:

    set /SP/services/servicetag state=enabled