9 Troubleshooting OSM Installation Problems

This chapter describes some of the issues you may encounter during the Oracle Communications Order and Service Management (OSM) installation process, and their solutions.

Coherence Configuration Error: ORA-00001: unique constraint

The following error can occur in the OSM WebLogic server console when creating an order:

com.mslv.oms.handler.InternalErrorException: ORA-00001: unique constraint ORDERMGMT4701.XPKOM_ORDER_FLOW_COORDINATOR) violated

This error occurs because incorrect or missing Coherence settings cause the nodes in the cluster to be unaware of each other. The servers are unaware that they must generate order IDs that take the other servers into consideration. This problem does not occur if the same server gets all of the createOrder requests. The problem occurs when any other server gets a request and uses the wrong formula to generate the order ID.

For more information about Coherence, see "Configuring Oracle Coherence for an OSM Cluster."

Coherence Configuration Error: [STUCK] ExecuteThread

The following thread error can occur in the OSM WebLogic server console when running an order:

[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@5d7fc269 WAITING

The osm-invocation and osm-distributed thread-count values are set too low. See "Configuring and Monitoring Coherence Threads" for information about increasing these settings.

Error About T3 After Initial OSM Startup

The first time you start the OSM server after installation, you may see an exception indicating T3 file attachment not found.

If this occurs, restart the server.

Node Manager Does Not Create IP Address for Whole Server Migration

When you start up a managed server that is configured for whole server migration, the managed server fails to start because node manager does not create the floating IP address for the managed server.

If this occurs, ensure that you have selected Automatic Server Migration Enabled when you configured the managed server. Node manager does not allocate IP addresses to managed server unless this value is selected. See "Configure Managed Servers for Whole Server Migration" for information about setting this value.

Database Connection Problems During Installation

If you receive database connection errors when clicking Connect on the Database Server Connection Information dialog box, click Retry. This sometimes fixes the problem.

This issue is related to latency in the database connection. Acceptable network latency should be between 0.2 and 0.4 msec. Anything higher than 1 msec can substantially reduce OSM performance.

To verify network latency, do the following:

  1. Log in to the machine running the OSM server.

  2. Run the following command:

    #ping -s osm_database
    

    where osm_database is the host name or IP address of the machine running the OSM database server.

    The system responds with lines similar to the following:

    PING osm_database: 56 data bytes
    64 bytes from osm_database: icmp_seq=0. time=0.389 ms
    64 bytes from osm_database: icmp_seq=1. time=0.357 ms
    

    A value for time of less than 0.4 indicates acceptable network latency. A value greater than 1.0 indicates excessive network latency.

    Note:

    Solaris uses the -s option. Linux and AIX do not require this option.

DataDictionary Expansion Level

If you are having issues deploying cartridges, the cause may be related to the DataDictionary expansion level. In Oracle Communications Design Studio, under Windows preferences, increase the DataDictionary expansion level to 10. In some cases, you may need to increase the level to more than 10.

JMS Server Connection Problems

After installation, when you restart the server, you may receive an error message from the JMS server connecting to the database. Many retries of the operation occur.

First, check the database connectivity as the database listener or database instance might be down. As a last resort, you may have to re-create the JMS server resource (not recommended) or re-run the OSM installation.

Problems When Running Multiple WebLogic Domains on One Host

If you are running multiple WebLogic domains on one host and you see errors such as web clients failing to load, JSP errors, or errors indicating that JSP pages can't be recompiled, you may not have set umask values properly to protect files in one WebLogic instance from other WebLogic instances.

JDBC Errors When First Order Submitted

If you receive JDBC errors when the first order is submitted to OSM, you may need to turn on JDBC logging. Refer to the Oracle WebLogic Server documentation.

Unable to Bring Up Managed Server After Database Failure

OSM JDBC data sources support global transactions through the logging last resource (LLR) WebLogic optimization. LLR transaction records must be available to resolve in-doubt transactions during recovery, which runs automatically at server startup. WebLogic does not start if the LLR table is unreachable. OSM also fails to start if the Oracle database is unreachable. In addition, if the database is Oracle RAC, both database nodes specified by the data source URLs must be reachable. When WebLogic Server starts, failure of a database node is handled gracefully (processing fails over to the other node).

Slow Query Performance in OSM Web Clients

If you have many cartridges deployed in OSM and have other hardware issues that may be impacting performance, the query performance of OSM web clients might be very slow.

To avoid slow query performance in OSM web clients, you can try setting cursor_sharing=FORCE as follows:

Caution:

Using the cursor_sharing=FORCE setting must be done with caution and is not recommended for all OSM installations (only those installations with specific hardware issues and that have a low number of OSM instances).
  1. Change cursor_sharing=FORCE

    Log into SQL*Plus as a user with ALTER SYSTEM privileges and run:

    alter system set cursor_sharing=FORCE scope=both;
    
  2. Restart OSM.

  3. Log in to the OSM web client.

    Expect performance to be slow the first time.

  4. For the Task web client, refresh the Worklist a few times.

    Response time should be much better.

  5. Log out of the OSM web client and log in again.

  6. Check that the query operation runs faster.

Delayed JMS Messages

Clock synchronization issues may cause JMS request and response messages to remain in a queue in a delayed state.

When a message is sent, it is stamped with the time on the sender's machine. When the message arrives at the JMS destination, if the recipient's machine has a clock that is running several minutes slower, the timestamp is displayed as a future time and the WebLogic server decides to delay the message until the future time arrives.

To prevent this problem, use network time protocol (NTP) servers to synchronize clocks across all machines in a cluster.

The short term work around for this issue is to manually move the message to another JMS server on a machine with a clock that is synchronized.

To monitor JMS messages in the OSM WebLogic server console:

  1. Log in to the OSM WebLogic server.

    The WebLogic Administration Console is displayed.

  2. Click Services.

  3. Select Messaging.

  4. Select JMSModules.

  5. Select the JMS module associated with the delayed message.

  6. Select the queue associated with the delayed message.

  7. Click the Monitoring tab.

  8. Select a destination to find the delayed message.

    Note:

    If the message is not in the first destination, check the second destination.
  9. Select Show messages.

  10. Select the message in the delayed state.

  11. Click Move.

Error Message For Events From a JMS Topic in a Cluster

The following message can occur when attempting to process an event from a JMS topic:

<Message-Driven EJB: YourCartridgeName_1.0.0.0.0_YourPluginName_orderCompleteEventMDB's transaction was rolled back. The transaction details are: ....

OSM does not support JMS topics within an OSM clustered environment. For more information about OSM queue configuration, see "OSM Integration with External Systems."

JMS Message Delivery Failure

If OSM drops and does not process JMS messages, ensure that the messages are not using uniform distributed queue (UDQ) format. OSM supports only weighted distributed queues. For more information about OSM queue configuration, see "OSM Integration with External Systems."

OSM and RCU Installers Are Slow to Run Database Tablespace Query

It can take an unusually long time for the OSM Installer and RCU Installer to run a database tablespace query. Purging the Oracle Database recycle bin ensures that the installers can run the database tablespace query more quickly.

To purge the Oracle Database recycle bin system wide:

  1. Log in to SQL*Plus as a user with sysdba privileges.

  2. Enter the following command:

    purge dba_recyclebin;
    

    The recycle bin is purged system wide.

To purge the Oracle Database recycle bin for a single user:

  1. Log in to SQL*Plus as the OSM installer database user.

  2. Enter the following command:

    purge recyclebin;
    

    The recycle bin for the database user is purged.

OSM Installer Issues for UNIX or Linux

If you are re-using the installation directory or the operating system temporary directory, you might have a problem running the installer on a UNIX or Linux system. To resolve this, you must start two sessions: a session for WebLogic Server, and separate a session for running OSM Installer.

To start a UNIX or Linux session for WebLogic Server:

  1. Ensure the OSM WebLogic Server has the following Java Virtual Machine (JVM) setting:

    -Djava.io.tmpdir=./tmp

    Note:

    If OSM WebLogic Server is installed in a cluster environment, create a tmp_servername directory in the WebLogic Server domain directory, and use the following setting for each server in the cluster:

    -Djava.io.tmpdir=./tmp_servername

    where servername is the name of each server in the cluster. For example, to install OSM on oms_cluster, which consists of oms_server1 and oms_server2, start oms_server1 by using -Djava.io.tmpdir=./tmp_oms_server1, and start oms_server2 by using -Djava.io.tmpdir=./tmp_oms_server2.

To start a UNIX or Linux session for running OSM Installer:

  1. Create a new directory, and then create a temporary directory within that new directory. For example:

    /home/osm_user/new_osm_install

    /home/osm_user/new_osm_install/tmp

  2. Download the OSM software media pack for your operating system from the Oracle software delivery website, located at:

    https://edelivery.oracle.com/

    and save it to the directory that you created. For example, /home/osm_user/new_osm_install.

  3. Ensure that the Oracle Database and Oracle WebLogic Server instances that you intend to use for OSM are running.

  4. Follow step 6 of the procedure to install OSM, located in the "Installing OSM" topic.

  5. In the temporary directory that contains the installation executable file, for example, /home/osm_user/new_osm_install, run the OSM Installer by doing one of the following:

    • On Oracle Linux and Red Hat Enterprise Linux, run the following command:

      InstallLinux.bin -is:tempdir ./tmp
      
    • On Oracle Solaris, run the following command:

      InstallSparc.bin -is:tempdir ./tmp
      
    • On IBM AIX, run the following command:

      InstallAix.bin -is:tempdir ./tmp
      
    • On HP-UX Itanium, run the following command:

      InstallHpuxItanium.bin -is:tempdir ./tmp
      
    • On Windows, navigate to the directory containing the installation executable files and double-click Install.exe.

  6. Continue with step 8 of the procedure to install OSM, located in the "Installing OSM" topic.

Command for unpack.jar Fails with a Write Error

If you run the unpack.jar command and you receive a write error, you must provide a target application tag (-app_dir) while running the command.

For example:

./unpack.sh -template=/scratch/oracle/Middleware/user_projects/domains/osmprak_72251to730_upgddomain_final8may.jar -domain=/scratch/oracle/Middleware/user_projects/domains/osmprak_72251to730_upgddomain -app_dir=/scratch/oracle/Middleware/user_projects/applications/osmprak_72251to730_upgddomain

JBoss Cache Timeouts

Long full garbage collections can cause JBoss timeout errors to appear in the log files.