Monitoring and Managing OSM

9 Monitoring and Managing OSM

This chapter describes how to monitor and manage the Oracle Communications Order and Service Management (OSM) system using the Oracle WebLogic Server Administration Console.

This chapter provides guidelines and best practices for monitoring an OSM deployment. This includes functional monitoring of particular orders or processes, and performance monitoring to assist in tuning. In order to effectively monitor OSM, you require a broad knowledge of many components, such as the OSM Managed Server, the Java Virtual Machine (JVM), the WebLogic Server, and the Oracle Database.

About Monitoring and Managing OSM

Many OSM monitoring tasks must be performed on various schedules. Some tasks should be performed daily, whereas others can be done weekly or even monthly. Many tasks can be done automatically, by configuring thresholds that raise warnings when the thresholds are exceeded. The output of many of the tasks can be interpreted as snapshots, whereas others should be interpreted only in the context of a series of data.

For each monitoring task in this chapter, there is a description of the item that the task monitors, why the item should be monitored, what tool(s) you should use to monitor the item, and information about what to monitor, including guidance on selecting values, such as thresholds.

The monitoring tasks are grouped into monitoring the application, monitoring input/output (I/O), and monitoring the host.

Accessing the WebLogic Server Administration Console

The WebLogic Server Administration Console is a web-based application that allows system administrators, support staff, and others to view the OSM application and associated configuration remotely. In OSM cloud native, use WebLogic Server Administration Console to verify the configuration that has been applied and to see that the OSM application is in a good state.

To access the Oracle WebLogic Server Administration console do one of the following:

If you are not connecting via Secure Socket Layer (SSL), enter the following URL into your browser:
```
http://admin.host:port/console
```
The base hostname required to access this instance is instance.project.osm.org. See "Planning and Validating Your Cloud Environment" in OSM Cloud Native Deployment Guide for details about hostname resolution.
If you are connecting via SSL, enter the following URL into your browser:
```
https://admin.host:SSLport/console
```

When started, the WebLogic Server Administration Console prompts for a password. This should be the password for a user that is a member of the Administrators group in WebLogic. One such user is the WebLogic administration user that was created when the domain was created. By default, the name of this user is weblogic.

After you have successfully logged in, the WebLogic console Home window is displayed.

Using the WebLogic Console to Determine the Status of the OSM Application

After you have logged into the WebLogic console, you can access information about the status of the WebLogic servers and OSM deployments.

To access the status of the OSM server and deployments:

Log in to the WebLogic Server Administration Console.
In the Domain Structure tree, expand Environment, and then click Servers.

The Summary of Servers window is displayed. Server statuses are contained in the Health column of the table.
In the Domain Structure tree, click Deployments.

The Summary of Deployments window is displayed. Deployment statuses are contained in the Health column of the table.

Refreshing OSM Metadata

When you are working with OSM, you may come across instructions to refresh the OSM metadata. There are different ways to do this. You can use whichever of them is easiest for you, as long as you meet any prerequisites.

To refresh the OSM metadata, do any one of the following:

If you have access to the Order Management web client, click Refresh Server Cache in the Administration area.
Scale up the number of managed servers.

Monitoring and Analyzing Performance

This section describes several tools that are specific to OSM, as well as some general tools, that gather data when your system has performance-related issues.

Monitoring Performance Using WebLogic Server Administration Console

The WebLogic Server Administration Console provides a real-time view of system performance.

Note:

For more information about tuning OSM production systems, see OSM Installation Guide.

To access the performance monitor:

Start the WebLogic Server Administration Console.
Click Environment, then Servers, and then select a server from the list.

This displays the General Configuration page for the selected server.
Click Monitoring.
Click the Health tab to view the health status for all OSM related sub-systems. If the status is not OK, review the reason and, if required, access the server log for more information.

Health status severity levels are shown in the bottom left pane of the console under System Status. (OK, Warn, Overloaded, Critical, Failed)
Click the Performance tab to view JVM memory utilization statistics for the server.

Refresh the screen several times to sample values for Health Free Percent. If memory usage remains elevated (for example, not falling to 50% or less while the system is under load), analyze the garbage collection logs and follow the recommendations described in the OSM Memory Tuning Guidelines (Doc ID: 2028249.1) knowledge article on My Oracle Support. For example, undeploy cartridges that are no longer needed or add a new managed server to the WebLogic Server cluster.
Click the Threads tab to monitor thread activity for the server. An important column to monitor is Pending Requests. A count of zero is optimal, meaning no user requests are stuck or waiting to be processed.
Click the Workload tab to monitor the Work Managers configured for the server.

Check Deferred Requests in the Max Threads Constraint table. If Deferred Requests does not go back to 0, you may need to tune the system (for example, add a new managed server to the WebLogic Server cluster or allocate more threads to OSM work managers).
Click the JTA tab to monitor transaction activity on the server. Ideally, there should be no Roll Back statistics in the summary; if so, view the server log file for more information.

Monitoring the Managed Server

The high-level analytical tools and considerations for monitoring the managed server include the following:

In the Weblogic administration console, ensure all are servers are marked as OK in system status. If the server(s) listed have another status (for example, Overloaded or Critical) you must investigate immediately to determine the particular sub-system of the server that is under stress. Also, check the server logs.
Use the Reporting page in OSM Task web client to monitor the orders that are in progress and task statistics. For information about the Reporting page, see OSM Task Web Client User's Guide. You can also analyze OSM order completion rates using a script. For more information, see "Tools for Performance Testing, Tuning, and Troubleshooting".

Analyzing Garbage Collection

Garbage collection logs for the performance test runs include important information about the heap usage for the particular test. Use this data for later analysis and adjustment of the JVM heap and for tuning garbage collection. Use GCViewer to analyze garbage collection data that you have collected using Oracle JDK. For more information about GCViewer, see "GCViewer".

Monitoring the Database

When you monitor the operating system on the database server, make sure the system has enough idle CPU and Input/Output (I/O) wait times are low (nothing more than few percent of I/O waits are acceptable).

You can monitor the database by using Oracle Enterprise Manager, a web-based tool that allows you to manage Oracle software. For more information about Enterprise Manager, see Oracle Enterprise Manager Cloud Control documentation.

Managing Logs

For details about managing logs in OSM cloud native, see "Exploring Alternate Configuration Options" in OSM Cloud Native Deployment Guide.

Secure vs Non-Secure Log Filtering

A stack trace is a list of the method calls that an application is in the middle of at the time an exception is thrown. Running OSM with stack trace logging enabled can be important for debugging an application during runtime.

Using the oms-config.xml parameter, enable_log_stacktraces, you can enable or disable logging stack traces. By default, this parameter is enabled. If there is a security concern about having log stack traces enabled, you can disable this parameter. For more information about setting this parameter, see "Configuring OSM with oms-config.xml".

Managing Database Connections

In OSM, database connections are managed through database pools that are set up in WebLogic.

The database pool connections can be configured through the shape specification file. See "Working with Shapes" in OSM Cloud Native Deployment Guide.

Using JMS Queues to Send Messages

OSM uses JMS Queues and Topics, which are both JMS Destinations. Queues follow a point-to-point communication paradigm, while Topics follow the publish and subscribe paradigm.

Note:

In an OSM clustered environment, you must use JMS queues as a JMS destination to receive JMS events. Do not use JMS topics in an OSM clustered environment.

When OSM sends data to an external system, such as UIM or ASAP, it does so by sending JMS messages to the appropriate JMS request queue of an external system.

If the external system is not processing the requests from OSM, the queues get backlogged. It is important to be able to monitor the size of the JMS queues in order to know whether or not they are backing up.

To monitor the JMS queues:

Login to the WebLogic Administration Console

Click Services/Messaging/JMS Servers/oms_jms_server.

The General Configuration page is displayed.
Click the Monitoring tab and then click Active Destinations.

A list of active destinations targeted to the server is displayed.

Note:

The default view of the table does not contain the Consumers Current column. We recommend that you customize the table using the Customize link to include this column, along with any other customizations you may want to make.

The Consumers Column column defines the current number of listeners on the destination. If a destination does not have any listeners, the external system does not receive the messages.

The Messages Current column defines the current number of unprocessed messages in the JMS destination. A large number of messages (for example, 10,000) in this destination is a problem. It means that the system is not keeping up, that the messages are not getting processed, or that the messages are getting processed but errors are occurring and the messages are getting put back on the destination.

OSM has the following JMS destinations present:

oms_behavior_queue: Used for customizing task assignment
oms_events: Internal destination used for events such as automation, notifications, and task state changes
oms_order_events: Used for order state changes such as OrderCreateEvent, OrderStateChangeEvent, AmendmentStartedEvent, OrderCancelledEvent
oms_order_updates: Internal destination used for processing amendments
oms_signal_topic: Internal destination used to trigger a metadata refresh

Sending Data to External Systems Using Plug-Ins

If there are external systems deployed to the same WebLogic instance as OSM, when you monitor the JMS destinations, watch for the following.

Note:

The important columns are Consumers, Messages, and Messages Received.

If the number in the messages column for these queues continues to grow, the external system may not be processing the messages sent by OSM. You must check to see if the external system is working properly.

If the number of consumers for the queues is 0, the external system may not have configured its listeners properly. Check to see if the external system is configured properly.

About OSM and XA Support

The OSM database does not support XA transactions because the Oracle thin-client driver used for JDBC connections does not support XA. However, the OSM WebLogic Server configuration uses an XA emulation feature in order to get a two-phase commit across JMS/JDBC automation transactions.

Even though OSM uses a non-XA driver for database transactions, external XA resources can still participate in transactions. For example, JMS bridges can be XA-enabled for an outside application, but the OSM side of the transaction still uses the non-XA emulated two-phase commit. Note that this also applies to JMS queues that support Application Integration Architecture (AIA) cartridges.

Using Work Managers to Prioritize Work

In OSM cloud native, work managers are configured as part of defining a shape. WebLogic Server uses a single thread pool in which all types of work are executed. Work managers allow you to define rules and run-time metrics that WebLogic Server uses to prioritize this work.

For more information about using work managers to optimize scheduled work, see "Using Work Managers to Optimize Scheduled Work" in Oracle Fusion Middleware Administering Server Environments for Oracle WebLogic Server.

Default Work Managers and Components

This section list the default work managers and components in an OSM instance.

Table 9-1 Default Work Managers and Components

Work Manager or Component	Configuration Properties	Description
Maximum Threads Constraint	Name: osmAutomationMaxThreadConstraint	Used by the work manager osmAutomationWorkManager to reserve a separate pool of threads for core order processing.
Maximum Threads Constraint	Name: osmGuiMaxThreadConstraint	Used by the work managers osmTaskClientWorkManager and osmOmClientWorkManager to reserve a separate pool of threads for the web clients.
Maximum Threads Constraint	Name: osmGuiMinThreadConstraint	Used by the work managers osmTaskClientWorkManager and osmOmClientWorkManager to guarantee that a minimum number of threads are reserved for the web clients.
Maximum Threads Constraint	Name: osmHttpApiMaxThreadConstraint	Used by the work managers osmWsHttpWorkManager and osmXmlWorkManager to reserve a separate pool of threads for APIs invoked over http.
Maximum Threads Constraint	Name: osmJmsApiMaxThreadConstraint	Used by the work manager osmWsJmsWorkManager to reserve a separate pool of threads for the WS/JMS API
Work Manager	Name: osmAutomationWorkManager Maximum Threads Constraint: osmAutomationMaxThreadConstraint	Used to process work performed by automation tasks.
Work Manager	Name: osmTaskClientWorkManager Maximum Threads Constraint: osmGuiMaxThreadConstraint Minimum Threads Constraint: osmGuiMinThreadConstraint	Used to process requests from manual users using the Task web client.
Work Manager	Name: osmWsJmsWorkManager Maximum Threads Constraint: osmJmsApiMaxThreadConstraint	Used to process OSM JMS Web Service requests.
Work Manager	Name: osmWsHttpWorkManager Maximum Threads Constraint: osmHttpApiMaxThreadConstraint	Used to process OSM HTTP Web Service requests.
Work Manager	Name: osmOmClientWorkManager Maximum Threads Constraint: osmGuiMaxThreadConstraint Minimum Threads Constraint: osmGuiMinThreadConstraint	Used to process requests from manual users using the Order Management web client.
Work Manager	Name: osmXmlWorkManager Maximum Threads Constraint: osmHttpApiMaxThreadConstraint Minimum Threads Constraint: Not configured	Used to process requests coming from external clients for the OSM XML API.

For details on configuring these values, see "Working with Shapes" in OSM Cloud Native Deployment Guide.

Overriding the Internet Explorer Language in the OMS Web Clients

If the Internet Explorer installation that is used to access the OMS web clients is set to a language other than English, and this language matches one of the properties files included in the oms.ear file, the web client prompts are displayed in the non-English language.

The language used in the web clients is controlled by the resources.properties file. Additional language property files in the oms.ear file include:

resources_cs.properties (Czech language properties file)
resources_zh.properties (Chinese language properties file)

To remove support for a non-English language, unpack the oms.ear file, remove the corresponding properties file, repack the oms.ear file. For more information about working with oms.ear file, see OSM Developer's Guide.

For example, if the browser is set to use the Czech language, and the resources_cs.properties file exists in the oms.ear file, the web client prompts are displayed in Czech. Removing the resources_cs.properties file causes the web client prompts to be displayed in English, even though the browser language setting is still configured to the Czech language.