|Oracle® Communications Order and Service Management Installation Guide
|PDF · Mobi · ePub|
This chapter describes Oracle Communications Order and Service Management (OSM) post-installation tasks.
The following sections describe general post-installation tasks.
If you installed OSM in a clustered environment, you must configure a T3 file path for each managed server in the cluster. This is required to support attachments for tasks and notifications when you work in the Task Web client.
To configure the T3 file path:
Locate the path in a shared drive that is accessible to all cluster members.
Create a directory, for example, /sharedisk/Attachments.
From each host replace domain_home/Attachments with a link to the /sharedisk/Attachments directory.
After installation, you may want to change OSM runtime parameters. For example, you can increase the number of rows in a worklist, or update the OSM administrator's e-mail address.
OSM runtime parameters are configured in the oms-config.xml file.
For details, see the chapter describing the oms-config.xml file in OSM System Administrator's Guide.
To display graphical representations such as Gantt charts and orchestration plans on UNIX systems, OSM requires an Xserver such as Xvfb (X virtual framebuffer), an X11 server that is available for most UNIX platforms.
The following steps apply to Solaris. For other operating systems, refer to the respective product documentation.
Obtain and install Xvfb for your UNIX operating system.
Refer to your OS product documentation for specifics on Xvfb. On Solaris, Xvfb is included in the SUNWxwsrv package.
Start the Xvfb X server.
When starting Xvfb, disable host-based access control restrictions by providing the -ac option on the command line. For example:
Xvfb :1 -ac
Set the value of the DISPLAY environment variable to the host where you want the display to show, and the display number. Export the DISPLAY variable. For example, in the sh shell, type:
Note:This command assumes X server is running on display number 1 and on the same computer where WebLogic Server is running. If you have a different configuration, specify the IP address of the computer where the X server is running instead of localhost.
For convenience, you can place the Xvfb and DISPLAY setting commands in your user account configuration shell script or in the WebLogic Server startup script (startWebLogic.sh) so that they run when you start OSM.
If you are running on Solaris 5.10, add the following statement to the Java command line in the startWebLogic.sh file for your WebLogic server domain:
If the above statement is missing from the Java command line, the following error may occur:
-X10: fatal IO error 22 (invalid argument) on X server "localhost:
In addition, performance and stability issues may occur when OSM accesses the server's display.
If you are running multiple instances of OSM on the same UNIX system, set up a dedicated Xvfb server for each instance.
To set up dedicated Xvfb servers for multiple instances of OSM:
Start one instance of the Xvfb server on display 1:
Set the value of the DISPLAY environment variable in the first WebLogic server domain instance to 1:
Start the first WebLogic server domain instance.
Note:You must increment the Xvfb server display and the DISPLAY environment variable by one for each additional OSM instance. For example, the second instance would have a value of 2 for the Xvfb server display and the DISPLAY environment variable.
The following sections describe high availability post-installation tasks.
If OSM is running on a WebLogic managed server, the behaviorBuild.properties file must be configured to specify the URL of the administration server (for example, http://hostname:port).
From the system prompt, execute the command
In the WebLogic Administration Console, perform the following steps:
Undeploy the custom plug-in from the administration server.
Deploy the custom plug-in to the managed server.
Target the custom plug-in to the managed server.
Whole server migration is a recommended, high-availability best practice that is used to migrate a WebLogic server instance, and all of its services, to a different physical machine upon failure.
Before configuring server migration, ensure that your environment meets the requirements outlined in "WebLogic Server Migration".
This section provides high-level instructions for configuring automatic whole server migration. For full configuration details, see the whole server migration chapter in Oracle Fusion Middleware Using Clusters for Oracle WebLogic Server.
To configure automatic whole server migration:
Obtain floating IP addresses for each managed server that will have migration enabled.
If your OSM clustered environment includes a proxy server, ensure the WebLogicCluster parameter in the domain_home/apps/OracleProxy4_proxyserver_proxy/WEB-INF/web.xml file of the proxy plug-in includes all managed servers (where proxyserver is the proxy server name).
<init-param> <param-name>WebLogicCluster</param-name> <param-value>10.147.241.156:7003|10.147.241.157:7003|10.147.241.156:7005</para m-value> <description>This must be a list of the servers in the cluster, in the form server1:http_port:ssl_port|server2:http_port:ssl_port</description> </init-param>
After updating web.xml, redeploy the proxy plug-in.
Configure Node Manager.
Node Manager must be running and configured to allow server migration. See "Configuring Node Manager on all Machines in the Domain" for more information.
Ensure that Oracle Coherence is configured for all managed servers by updating the well-known-addresses in the tangosol-coherence.xml file.
See "About Configuring Coherence for an OSM Installation" for more information.
Determine the type of leasing to use, either non-database consensus leasing or high-availability database leasing. If you are using the database for leasing, you must configure the database for server migration.
Configure the server migration targets. Each server can have a different set of candidate machines, or they can all have the same set.
If a JDBC file store is configured, configure a custom persistent store that is targeted to the same migratable target as the server. Unless you are using pre/post migration scripts to move the store data across migrated servers, the store must be configured such that all the candidate servers in the migratable target have access to it after migration.
Test the server migration.
Restart the administration server.
If whole server migration is not possible, JMS service migration is an alternative. WebLogic Server supports service-level migration for JMS-related services, the JTA Transaction Recovery Service, and user-defined singleton services. JMS Service and JTA must be migrated together.
For configuration options, see the service migration chapter in Oracle Fusion Middleware Using Clusters for Oracle WebLogic Server. For detailed configuration instructions, see the WebLogic Administration Console Help.
To configure JMS-related service migration:
If using automatic service migration, configure the leasing basis for the cluster (either database or consensus).
Configure migratable targets for your JMS-related services. This includes specifying a preferred host server in the cluster, selecting candidate servers to use as a backup for services on the migratable target, and specifying a migration policy.
Note:It is recommended that you configure your JMS server as a migratable target with its policy set to Auto-Migrate Failure-Recovery Services. You should also set Restart On Failure to enable the failed service to be restarted in place first, instead of being migrated.
For persistent messaging, configure a custom persistent store that is targeted to the same migratable target as the JMS service. Unless you are using pre/post migration scripts to move the store data across migrated servers, the store must be configured such that all the candidate servers in the migratable target have access to it.
Target the JMS services to the same migratable target used by the custom persistent store.
This section describes how to configure a WebLogic instance to restart automatically when a database configured for cold cluster failover fails. For each server in the cluster, navigate to the Health Monitoring tab and set the following parameters:
Auto Kill if Failed - Select this parameter to enable Node Manager to automatically kill the server if its health state is failed.
Auto Restart - Select this parameter to enable Node Manager to automatically restart the failed server.
Restart Delay Seconds - Set this value to the number of seconds Node Manager should wait before attempting to restart the server. The value should allow ample time for the database failover to complete, for example, 300 seconds.
You may also want to consider increasing the JMS Redelivery Limit and possibly the JMS Redelivery Delay Override parameters to ensure that the JMS message redelivery limit is not exceeded during database failover. If the redelivery limit for a JMS message is exceeded, the message is normally delivered to a fallout queue and the symptom is a stuck order. See OSM System Administrator's Guide for more information about JMS redelivery configuration settings.
OSM supports high availability in the database layer through configuration of Oracle Real Application Clusters (RAC) for either:
Failover, also known as warm standby or active-passive Oracle RAC.
Load balancing, or active-active Oracle RAC.
Each WebLogic Server instance in the OSM cluster interacts with Oracle RAC through a WebLogic multi data source configured for failover with two data sources, one for each Oracle RAC instance. This setup is used for both active-passive and active-active configurations. In an active-active configuration, load balancing is achieved by evenly distributing the clustered server nodes into groups in an alternating fashion. See "Connecting Oracle RAC with JDBC Multi Data Source" for a discussion of how multi data sources and data sources are configured for Oracle RAC failover and load balancing.
During installation, you can allow the OSM installer to automatically create the appropriate configuration based on your selections and the database parameters of the Oracle RAC instances. You may also decide to manually configure one or more additional data sources after installation. See "Manually Creating and Configuring the Data Source" for details.
This section describes the steps required to manually create and configure additional data sources for an Oracle RAC instance. In an active-passive configuration you create one data source; in an active-active configuration you create two data sources.
To create and configure an additional data source for an Oracle RAC instance:
Log in to the WebLogic Administration Server Console.
In the Domain Structure tree, expand Services, JDBC, then select Data Sources.
On the Summary of JDBC Data Sources page, click New.
On the JDBC Data Sources Properties page, do the following:
Enter a name for the JDBC data source.
If you have an active-passive configuration, name the data source osm_pool_2.
If you have an active-active configuration, name the data sources osm_pool_rac2_group_a and osm_pool_rac2_group_b.
Enter a unique JNDI name for the data source in either configuration. For example, oracle/communications/osm/internal/jdbc/pool_2.
Select Oracle as the database type.
Select one of the following non-XA thin drivers:
If your Oracle RAC database is configured with local listeners, select *Oracle's Driver (Thin) for Instance connections; Versions: 9.0.1 and later.
If your Oracle RAC database is configured with a remote listener (the default), select *Oracle's Driver (Thin) for RAC Service-Instance connections; Versions: 10 and later.
On the Transaction Options page, do the following:
Ensure the default option Supports Global Transactions is selected.
Select the option Logging Last Resource.
On the Connection Properties page, specify the service name, database name, host, and port of the additional Oracle RAC instance based on one of the following scenarios:
If your Oracle RAC database is configured with a remote listener and server-side load balancing (the default), the host and port of the new data source must be the same as the host and port of the existing data source. Additionally, if the existing data source specifies a service name and the instance name of the Oracle RAC instance, the new data source must also specify the service name and instance name of the Oracle RAC instance. (The instance name is required in order to override server-side load balancing. See "Listener Considerations for Oracle RAC" for a full discussion of listener functionality.)
For example, if the existing data source URL is:
The new data source URL is:
If your Oracle RAC database is configured with local listeners, the host and port of the new data source must be different than the host and port of the existing data source. If the existing data source specifies a service name, the new data source must specify the same service name.
Enter the OSM database schema user name and password, and then confirm the password.
On the Test Database Connection page, review the connection parameters and click Test Configuration.
WebLogic attempts to create a connection from the administration server to the database. Results are displayed at the top of the page. If the test is unsuccessful, you should correct any configuration errors and retry the test.
If the JDBC driver you selected is not installed on the administration server, you should click Next to skip this step.
On the Select Targets page, do one of the following:
If you have an active-passive configuration, target the new data source to the entire cluster.
If you have an active-active configuration, target the data source to part of the cluster. For example, target osm_pool_rac2_group_a to osm1 and osm_pool_rac2_group_b to osm2.
Your configuration is saved and the data source is deployed to the cluster.
Configure the connection properties for the newly created data source. See "Configuring Connection Pool Properties".
Add the data source to the preconfigured multi data source. See "Adding the Data Source to the Multi Data Source".
In addition to the properties you defined in the previous section, you must configure the data source's connection pool properties.
To configure the connection pool properties, do the following:
Select the data source and navigate to the Connection Pool tab.
In the Properties list, enter oracle.net.CONNECT_TIMEOUT=10000.
In the Initial Capacity field, enter 0.
Click Advanced options.
Select the Test Connections on Reserve check box.
In the Test Frequency field, enter 300.
In the Test Table Name field, enter SQL SELECT 1 FROM DUAL.
In the Seconds to Trust an Idle Pool Connection, enter 10.
In the Shrink Frequency field, enter 900.
Save your changes.
You can make all data source and multi data source changes in a single edit session. You do not have to save your changes between steps.
To add the new data source to the multi data source:
In the Domain Structure tree, expand Services, then select Data Sources.
On the Summary of Data Sources page, click the multi data source name.
Click the Configuration Data Sources tab, and do one of the following:
For an active-passive configuration, move the data source that you want to include in the multi data source to the Chosen list.
For an active-active configuration, add the data sources to the multi data sources in the order specified:
Multi data source oms_pool_group_a must contain the two data sources osm_pool_rac1_group_a and osm_pool_rac2_group_a, in this order.
Multi data source oms_pool_group_b must contain the two data sources osm_pool_rac2_group_b and osm_pool_rac1_group_b, in this order.
Note:To create the correct alternating sequence required by the Oracle RAC database instances, the osm_pool_rac2_group_b data source must appear in the list before the osm_pool_rac1_group_b data source. See Figure B-5 for more information.
Note:Oracle recommends that you shut down all managed servers before changing the order of the data sources in the list because the first data source in the list is used to compose the name of the Oracle Coherence cluster when OSM starts.
Save your changes.
In addition to the distributed queues that OSM creates during installation, you may need to create and configure your own custom distributed queues to support your particular environment.
Use the following high-level steps to create and configure a distributed queue and its members:
Create the member queues, one per managed server pinned to each JMS server.
Create the distributed queue with equally-weighted member queues and target the distributed queue to the cluster.
Confirm that the distributed queue is accessible and visible to all managed servers in the cluster.
In the WebLogic Administration Console, expand Services, Messaging, and then select JMS Modules.
On the Summary of JMS Modules page, select oms_jms_module.
Note:You must use the predefined JMS system module to allow the connection factory to deliver messages to the correct managed server. Do not create a new JMS system module.
On the Configuration page, above the Summary of Resources table, click New.
On the Create a New JMS System Module Resource page, select Queue and click Next.
On the JMS Destination Properties page, enter the name and JNDI name of the member queue, and then click Next.
Select the subdeployment you want to assign to the queue.
The number of subdeployments should equal the number of JMS servers, and hence managed servers configured in the cluster.
Upon selection of a subdeployment, the queue is automatically assigned to the correct JMS server.
Click Finish. The newly created member queue is displayed in the resources list.
For information about using JMS queues, see OSM System Administrator's Guide.
On the Configuration page, above the Summary of Resources table, click New.
On the Create a New JMS System Module Resource page, select Distributed Queue and click Next.
On the JMS Distributed Destination Properties page, do the following:
Enter the name and JNDI name of the distributed queue.
Ensure the Load Balancing Policy is set to Round-Robin.
Deselect Allocate Members Uniformly.
From the Members Available list, select the newly created queues and move them to the Chosen list.
Click Finish. The distributed queue is displayed in the resources list.
The following sections describe OSM integration tasks between OSM, Oracle Communications ASAP, and Oracle Communications Unified Inventory Management (UIM). These integration tasks are also applicable for OSM integration with other remote applications.
Oracle recommends that you create a Store-and-Forward (SAF) agent and JMS bridge between the Oracle Communications ASAP WebLogic server or Oracle Communications IP Service Activator WebLogic server and the OSM WebLogic server. Oracle recommends this SAF agent and JMS bridge for the JMS or Web Service interfaces to ensure reliable communication.
Figure 9-1 illustrates an example SAF and JMS bridge configuration between the Web Service interface on ASAP and a Web Service client on OSM. An example SAF and JMS bridge configuration between the Web Service interface on IP Service Activator and a Web Service client on OSM would appear the same.
In this example, an OSM SAF agent sends requests to the ASAP request queue, and ASAP returns responses through the ASAP SAF agent to the OSM reply-to queue. In addition, ASAP sends work order state changes from the JSRP XVTEventTopic through a JMS bridge with a SAF agent to the OSM event queue.
For detailed instructions for creating SAF and JMS bridges between ASAP and OSM, see Configuring WebLogic Resources for OSM Integration With ASAP And UIM On Different Domains (Doc ID 1431235.1) knowledge article in Oracle Support at:
This article is also applicable to IP Service Activator or to any other remote application that uses a WebLogic JMS server to send and receive Web Service or JMS messages.
Note:When using the above instructions with any remote application other than ASAP, remember to change any reference to ASAP to the remote application for which you are creating bridges. For example, change ASAP to IPSA.
Oracle recommends that you create a SAF agent between the UIM WebLogic server and the OSM WebLogic server. Oracle recommends this SAF agent for the Web Service interfaces to ensure reliable communication.
Figure 9-2 illustrates an example SAF configuration between the Web Service interface on UIM and a Web Service client on OSM.
In this example, an OSM SAF agent sends requests to the UIM request queue, and UIM returns responses through the UIM SAF agent to the OSM reply-to queue.
For detailed instructions for creating SAF agents between UIM and OSM, see Configuring WebLogic Resources for OSM Integration With ASAP And UIM On Different Domains (Doc ID 1431235.1) knowledge article in Oracle Support at:
This article is applicable to any remote application that uses a WebLogic JMS server to send and receive Web Service messages.
This section describes some of the issues you may encounter during the OSM installation process, and their solutions.
If the OSM servers do not restart or restarts with errors after you installed OSM in a WebLogic cluster, the problem is often because of the Coherence configuration.
Do the following to check your Coherence configuration:
Open the WebLogic console output for the failing server and verify the following:
Make sure that Coherence is reads the file identified by startup property osm.coherence.cluster.config.override (see "About Configuring Coherence for an OSM Installation").
A console log message Oracle Coherence stating Loaded operational overrides from "file:path/Coherence_config"' (where path is the absolute path to the Coherence configuration file, and Conherence_config is the Coherence configuration file) indicates that the indicated file is read correctly.
A console log containing string Oracle Coherence stating Optional configuration override "path/Coherence_config" is not specified" indicates that the file could not be found.
Make sure that the values for tangosol.coherence.localhost and tangosol.coherence.localport correspond to one of the <socket-address/> entries in the configuration file (see "About Configuring Coherence for an OSM Installation"). If these values do not match a <socket-address/> entry in the configuration file, the OSM server may produce a timeout exception while waiting for the <socket-address/> listed in the <well-known-addresses/> section.
The following error can occur in the OSM WebLogic server console when creating an order:
com.mslv.oms.handler.InternalErrorException: ORA-00001: unique constraint ORDERMGMT4701.XPKOM_ORDER_FLOW_COORDINATOR) violated
This error occurs because incorrect or missing Coherence settings cause the nodes in the cluster to be unaware of each other. The servers are unaware that they need to generate order IDs that take the other servers into consideration. This problem does not occur if the same server gets all of the createOrder requests. The problem occurs when any other server gets a request and uses the wrong formula to generate the order ID.
For more information about Coherence, see "Configuring Oracle Coherence for an OSM Cluster".
The following thread error can occur in the OSM WebLogic server console when running an order:
[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@5d7fc269 WAITING
The thread count settings for Coherence parameters is not set correctly. Increase all thread count settings from 0 to 10, including:
You can add these parameters to the Coherence parameter file (see About Configuring Coherence for an OSM Installation).
<init-param id="1"> <param-name>thread-count</param-name> <param-value system-property="tangosol.coherence.invocation.threads">10</param-value> </init-param>
To monitor threads in the OSM WebLogic server console:
Log into the OSM WebLogic server.
The WebLogic Administration Console is displayed.
The Summary of Servers is displayed.
Select any OSM server.
Click the Monitoring tab.
Click the Threads sub-tab.
The Self-Tuning Thread Pool Threads table is displayed listing the names of the threads associated to the server.
If you have timeout issues when deploying cartridges, the problem may be caused by incorrect WebLogic parameters.
If you receive JDBC timeout exceptions when deploying large cartridges, you may need to update the transaction timeout parameter in the WebLogic server startup script, the timeout seconds Java Transaction API (JTA) parameter, or the Max Stuck Thread Time parameter. See "Modifying Deployment Parameters".
The first time you start the OSM server after installation, you may see an exception indicating T3 file attachment not found.
If this occurs, restart the server.
If you receive database connection errors when clicking Connect on the Database Server Connection Information dialog box, click Retry. This sometimes fixes the problem.
This issue is related to latency in the database connection. Acceptable network latency should be between 0.2 and 0.4 msec. Anything higher than 1 msec can substantially reduce OSM performance.
To verify network latency, do the following:
Login to the machine running the OSM server.
Run the following command:
#ping -s osm_database
where osm_database or is the host name or IP address of the machine running the OSM database server.
The system responds with lines similar to the following:
PING osm_database: 56 data bytes 64 bytes from osm_database: icmp_seq=0. time=0.389 ms 64 bytes from osm_database: icmp_seq=1. time=0.357 ms
A value for time of less than 0.4 indicates acceptable network latency. A value greater than 1.0 indicates excessive network latency. For more information about optimizing OSM hardware configurations to minimize network latency, see "Performance Recommendations".
Note:Solaris uses the -s option. Linux and AIX do not require this option.
If you are having issues deploying cartridges, the cause may be related to the DataDictionary expansion level. In Oracle Communications Design Studio, under Windows preferences, increase the DataDictionary expansion level to 10. In some cases, you may need to increase the level to more than 10.
After installation, when you restart the server, you may receive an error message from the JMS server connecting to the database. Many retries of the operation occur.
First, check the database connectivity as the database listener or database instance might be down. As a last resort, you may have to re-create the JMS server resource (not recommended) or re-run the OSM installation.
If you are running multiple WebLogic domains on one host and you see errors such as Web clients failing to load, JSP errors, or errors indicating that JSP pages can't be recompiled, you may not have set umask values properly to protect files in one WebLogic instance from other WebLogic instances.
If you receive JDBC errors when the first order is submitted to OSM, you may need to turn on JDBC logging. Refer to the Oracle WebLogic Server documentation.
OSM JDBC data sources support global transactions through the logging last resource (LLR) WebLogic optimization. LLR transaction records must be available to resolve in-doubt transactions during recovery, which runs automatically at server startup. WebLogic does not start if the LLR table is unreachable. OSM also fails to start if the Oracle database is unreachable. In addition, if the database is Oracle RAC, both database nodes specified by the data source URLs must be reachable. When Weblogic starts, failure of a database node is handled gracefully (processing fails over to the other node).
If you have many cartridges deployed in OSM and have other hardware issues that may be impacting performance, the query performance of OSM Web clients might be very slow.
To avoid slow query performance in OSM Web clients, you can try setting cursor_sharing=FORCE as follows:
Caution:Using the cursor_sharing=FORCE setting must be done with caution and is not recommended for all OSM installations (only those installations with specific hardware issues and that have a low number of OSM instances).
Log into SQL*Plus as a user with ALTER SYSTEM privileges and run:
alter system set cursor_sharing=FORCE scope=both;
Log in to the OSM Web client.
Expect performance to be slow the first time.
For the Task Web client, refresh the Worklist a few times.
Response time should be much better.
Log out of the OSM Web client and log in again.
Check that the query operation runs faster.
Clock synchronization issues may cause JMS request and response messages to remain in a queue in a delayed state.
When a message is sent, it is stamped with the time on the sender's machine. When the message arrives at the JMS destination, if the recipient's machine has a clock that is running several minutes slower, the timestamp is displayed as a future time and the WebLogic server decides to delay the message until the future time arrives.
To prevent this problem, use network time protocol (NTP) servers to synchronize clocks across all machines in a cluster.
The short term work around for this issue is to manually move the message to another JMS server on a machine with a clock that is synchronized.
To monitor threads in the OSM WebLogic server console:
Log in to the OSM WebLogic server.
The WebLogic Administration Console is displayed.
Select the JMS module associated with the delayed message.
Select the queue associated with the delayed message.
Click the Monitoring tab.
Select a destination to find the delayed message.
Note:If the message is not in the first destination, check the second destination.
Select Show messages.
Select the message in the delayed state.
The following message can occur when attempting to process an event from a JMS topic:
<Message-Driven EJB: YourCartridgeName_18.104.22.168.0_YourPluginName_orderCompleteEventMDB's transaction was rolled back. The transaction details are: ....
OSM does not support JMS topics within an OSM clustered environment. For more information about OSM queue configuration, see "OSM Integration Post-Installation Tasks".
If OSM drops and does not process JMS messages, make sure that the messages are not using uniform distributed queue (UDQ) format. OSM supports only weighted distributed queues. For more information about OSM queue configuration, see "OSM Integration Post-Installation Tasks".