Monitoring and Managing OSM

10 Monitoring and Managing OSM

This chapter describes how to monitor and manage the Oracle Communications Order and Service Management (OSM) system using the Oracle WebLogic Server Administration Console.

This chapter provides guidelines and best practices for monitoring an OSM deployment. This includes functional monitoring of particular orders or processes, and performance monitoring to assist in tuning. In order to effectively monitor OSM, you require a broad knowledge of many components, such as the OSM Managed Server, the Java Virtual Machine (JVM), the WebLogic Server, and the Oracle Database.

About Monitoring and Managing OSM

Many OSM monitoring tasks must be performed on various schedules. Some tasks should be performed daily, whereas others can be done weekly or even monthly. Many tasks can be done automatically, by configuring thresholds that raise warnings when the thresholds are exceeded. The output of many of the tasks can be interpreted as snapshots, whereas others should be interpreted only in the context of a series of data.

For each monitoring task in this chapter, there is a description of the item that the task monitors, why the item should be monitored, what tool(s) you should use to monitor the item, and information about what to monitor, including guidance on selecting values, such as thresholds.

The monitoring tasks are grouped into monitoring the application, monitoring input/output (I/O), and monitoring the host.

About Monitoring OSM Using WebLogic Server Administration Console

Oracle WebLogic Server requires a set of interrelated resources, such as database connections, Java Messaging Service (JMS) queues, execution threads, transactions, and system memory to work together in order to provide the functionality required of OSM.

You use the WebLogic Server Administration Console to manage these resources, including tasks such as starting and stopping servers, balancing the load on servers or connection pools, selecting and monitoring the configuration of resources, detecting and correcting problems, monitoring and evaluating system performance, and making sure that OSM is correctly deployed to the target servers.

The WebLogic Server Administration Console is a web-based application that allows system administrators, support staff, and others to monitor and manage the OSM application remotely.

See the Oracle WebLogic documentation for more information.

Accessing the WebLogic Server Administration Console

To access the Oracle WebLogic Server Administration console do one of the following:

If you are not connecting via Secure Socket Layer (SSL), enter the following URL into your browser:
```
http://host:port/console
```
where host is the DNS name or IP address of the computer on which the Administration server is installed, and port is the port on which the Administration server is listening for requests.
If you are connecting via SSL, enter the following URL into your browser:
```
https://host:SSLport/console
```
where host is the DNS name or IP address of the computer on which the Administration server is installed, and SSLport is the port on which the Administration server is listening for SSL requests. This is a different port from the one used for non-SSL requests.

When started, the WebLogic Server Administration Console prompts for a password. This should be the password for a user that is a member of the Administrators group in WebLogic. One such user is the WebLogic administration user that was created when the domain was created. By default, the name of this user is weblogic.

After you have successfully logged in, the WebLogic console Home window is displayed.

Using the WebLogic Console to Determine the Status of the OSM Application

After you have logged into the WebLogic console, you can access information about the status of the WebLogic servers and OSM deployments.

To access the status of the OSM server and deployments:

Log in to the WebLogic Server Administration Console.
In the Domain Structure tree, expand Environment, and then click Servers.

The Summary of Servers window is displayed. Server statuses are contained in the Health column of the table.
In the Domain Structure tree, click Deployments.

The Summary of Deployments window is displayed. Deployment statuses are contained in the Health column of the table.

Note:

If any of the deployments are not in the status that you expected, you can use the buttons on this window to start and stop individual deployments, if necessary.

Using the WebLogic Scripting Tool

Several OSM features use the WebLogic Scripting Tool (WLST). For security, Oracle recommends that you configure secure communication for the administration or managed server on which you want to run WLST.

To configure secure communication for WLST on the managed server:

Configure the managed server to use one of the following:
- Demo keystore
- Custom keystore
  
  Note:
  
  For information about configuring demo keystores, see Oracle Fusion Middleware Administering Security for Oracle WebLogic Server. For information about configuring custom keystores, see OSM Installation Guide for the topic about setting up secure HTTPS connections.

If you are using a demo keystore, set the CLASS_JMV_ARG, as in the following example:

export CONFIG_JVM_ARGS="-Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.TrustKeyStore=DemoTrust"

If you are using a custom keystore, set the CLASS_JMV_ARG, as in the following example:
```
export CONFIG_JVM_ARGS="-Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.TrustKeyStore=CustomTrust -Dweblogic.security.CustomTrustKeyStoreFileName=/OSM-QA/oracle/Middleware1213/user_projects/domains/OSM_CS_Cluster_4VM_6Nodes_2RAC/config/security/jks/osm.jks"
```
Note:

You need to add other parameters to the CLASS_JVM_ARGS according to different keystore and password requirements. For example, to use the trusted Certificate Authorities (CA) from another keystore, specify the following:
```
-Dweblogic.security.CustomTrustKeyStoreType=type
-Dweblogic.security.CustomTrustKeyStorePassPhrase=passphrase
```
where type is the default keystore type specified in the JDK java.security file.

Passphrase defaults to no passphrase. which is allowed by some keystore types to grant read-only access. Other keystore types always require a keystore passphrase.
To create a WLST secure communication session, run the connect command, as in following example:
```
connect('WLS_Admin_Username', 'WLS_Admin_Password', 't3s://hostname:port')
```
where hostname is the hostname of the WebLogic Server, and port is the SSL port number of the WebLogic Server.

Refreshing OSM Metadata

When you are working with OSM, you may come across instructions to refresh the OSM metadata. There are different ways to do this. You can use whichever of them is easiest for you, as long as you meet any prerequisites.

To refresh the OSM metadata, do any one of the following:

If you have access to the Order Management web client, click Refresh Server Cache in the Administration area.
If you are using a clustered WebLogic deployment, start a managed server that is running OSM.
If you have configured an external oms-config.xml file, touch the file.
Shut down and restart OSM.

Monitoring and Analyzing Performance

This section describes several tools that are specific to OSM, as well as some general tools, that gather data when your system has performance-related issues.

Monitoring Performance Using WebLogic Server Administration Console

The WebLogic Server Administration Console provides a real-time view of system performance.

Note:

For more information about tuning OSM production systems, see OSM Installation Guide.

To access the performance monitor:

Start the WebLogic Server Administration Console.
Click Environment, then Servers, and then select a server from the list.

This displays the General Configuration page for the selected server.
Click Monitoring.
Click the Health tab to view the health status for all OSM related sub-systems. If the status is not OK, review the reason and, if required, access the server log for more information.

Health status severity levels are shown in the bottom left pane of the console under System Status. (OK, Warn, Overloaded, Critical, Failed)
Click the Performance tab to view JVM memory utilization statistics for the server.

Refresh the screen several times to sample values for Health Free Percent. If memory usage remains elevated (for example, not falling to 50% or less while the system is under load), analyze the garbage collection logs and follow the recommendations described in the OSM Memory Tuning Guidelines (Doc ID: 2028249.1) knowledge article on My Oracle Support. For example, undeploy cartridges that are no longer needed or add a new managed server to the WebLogic Server cluster.
Click the Threads tab to monitor thread activity for the server. An important column to monitor is Pending Requests. A count of zero is optimal, meaning no user requests are stuck or waiting to be processed.
Click the Workload tab to monitor the Work Managers configured for the server.

Check Deferred Requests in the Max Threads Constraint table. If Deferred Requests does not go back to 0, you may need to tune the system (for example, add a new managed server to the WebLogic Server cluster or allocate more threads to OSM work managers).
Click the JTA tab to monitor transaction activity on the server. Ideally, there should be no Roll Back statistics in the summary; if so, view the server log file for more information.

Monitoring the Managed Server

The high-level analytical tools and considerations for monitoring the managed server include the following:

In the Weblogic administration console, ensure all are servers are marked as OK in system status. If the server(s) listed have another status (for example, Overloaded or Critical) you must investigate immediately to determine the particular sub-system of the server that is under stress. Also, check the server logs.
Use the Reporting page in OSM Task web client to monitor the orders that are in progress and task statistics. For information about the Reporting page, see OSM Task Web Client User's Guide. You can also analyze OSM order completion rates using a script. For more information, see "Tools for Performance Testing, Tuning, and Troubleshooting".

Analyzing Garbage Collection

Garbage collection logs for the performance test runs include important information about the heap usage for the particular test. Use this data for later analysis and adjustment of the JVM heap and for tuning garbage collection. Use GCViewer to analyze garbage collection data that you have collected using Oracle JDK. For more information about GCViewer, see "GCViewer".

Analyzing Class Loading

Class loading issues result in "class not found" errors. WebLogic, from version 10.3.4 onward, has the Classloader Analysis Tool (CAT) available by default. For more information about the tool, see the section about using the Classloader Analysis Tool in Oracle Fusion Middleware Developing Applications for Oracle WebLogic Server document.

Coherence Datagram Testing

OSM uses Coherence for caching, which works for clustered environments. You can determine, using the datagram test, whether Coherence issues are because of either network or configuration problems. For more information, see the section about running the datagram test utility in Oracle Coherence Administrator's Guide.

Monitoring WebLogic Server

Monitor the WebLogic Server by using either WLST or JMX. Resources are always consumed when using monitoring tools and Oracle recommends that you begin by using WLST, because it uses fewer resources compared to JMX. Add other tools only when required. For information about using WLST over a secure connection, see the security section of OSM System Administrator's Guide.

There are open source projects available for monitoring by using JMX calls from the server. These monitor WebLogic server activities, such as JMS queues, JDBC connections, execute thread pool, and so on.

The following link provides tools that allow you to monitor by using JMX:

https://github.com/romix/JMX-server-monitoring/

Monitoring the Operating System

The most important aspects to consider when monitoring the operating system are: CPU, memory, and network use.

CPU use on the system should not exceed 90 percent. If you must keep latency to a minimum on your system, keep CPU use even lower. Additionally, you must account for times of peak use, as well as the possibility of increased load on the remaining servers in case of a cluster node failure.

For memory use, configure your OSM system in order to avoid paging and swapping. For more information, see the JVM section of the Weblogic Server documentation.

For network use, bandwidth and latency are the primary factors that have an impact on OSM performance and throughput. In particular, OSM is sensitive to latency between Oracle database servers and WebLogic servers. Monitor the amount of data being transferred across the network by checking the data transferred between WebLogic servers, and between WebLogic servers and Oracle Database servers. This amount must not exceed your network bandwidth. You can check for symptoms of insufficient bandwidth by monitoring for retransmission and duplicate packets.

Monitor the WebLogic servers to ensure that I/O wait times are low.

Common operating system commands are available to measure and monitor CPU, memory, and network and storage performance and usage on Linux and Solaris systems. This includes top on Linux systems or prstat on Solaris operating systems, as well as sar, mpstat, iostat, netstat, vmstat, and ping on both platforms.

OSWatcher Black Box is a tool that monitors operating system statistics. For more information, see "OSW Black Box".

You can also use the Remote Diagnostics Agent (RDA) tool to generate a report that includes operating system measurements. For more information about this tool, see "Remote Diagnostics Agent".

Gathering OSM Execution Statistics

An execution statistics API allows you to gather execution statistics and provides a JMX interface for remote management and monitoring. The execution statistics MXBeans are registered with the Weblogic Server Runtime Bean Server.

To enable JConsole to access the MXBeans:

Enable the IIOP protocol for the WebLogic Server instance that hosts your MBeans.
Set the JAVA_HOME and WLS_HOME environment variables.

At the command prompt, enter the following commands:

UNIX/Linux:

jconsole -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:$WLS_HOME/lib/wljmxclient.jar -J-Djmx.remote.protocol.provider.pkgs=weblogic.management.remote service:jmx:rmi:///jndi/iiop://hostname:port/weblogic.management.mbeanservers.runtime

Windows:

jconsole -J-Djava.class.path=%JAVA_HOME%\lib\jconsole.jar;%JAVA_HOME%\lib\tools.jar;%WLS_HOME%\lib\wljmxclient.jar -J-Djmx.remote.protocol.provider.pkgs=weblogic.management.remote service:jmx:rmi:///jndi/iiop://localhost:7001/weblogic.management.mbeanservers.runtime

The New Connection screen opens.

Enter the user name and password for the WebLogic administrator.
Click the MBeans tab.
Expand the oracle.communications.ordermanagement node.

The registered execution statistics managers for each application are displayed under the ExecutionStatsManagerMXBean node.
In Operation Invocation, click Start.

Note:

If you want to export the statistics to an XML file, use the dump method by clicking dump. You can leave the argument empty (which it is by default) and the file that is created is named using the application ID. The file is then saved under the stats sub-directory of the WebLogic domain.

Analyzing Heap Dumps

You analyze the heap so you can find out what objects are using memory. Use a tool such as Eclipse Memory Analyzer (MAT) to analyze the heap dump. For documentation and to download the tool, use the following link:

http://www.eclipse.org/mat/

Note:

Use the following command in Linux and UNIX to produce a heap dump:

jmap -heap:format=live pid

where pid is the ID of the OSM WebLogic Server processor. For more information about heap dumps, see Oracle JMap documentation.

Analyzing Thread Dumps

A thread dump is a snapshot of the status of every thread within a JVM at a particular point. Analyzing thread dumps can be useful if your OSM system becomes sluggish or hangs. This type of analysis can reveal hot spots, in the form of frequently executed code, as well as threads that are stuck because of deadlocks or thread contention.

While a single thread dump can sometimes reveal the problem (for example, a thread deadlock), a series of thread dumps generated at regular intervals (for example, 10 thread dumps at 30-second intervals) is often required to troubleshoot more complex problems, like threads that are not progressing and are waiting for other processes.

You can use the VisualVM tool to generate and analyze thread dumps. The JStack tool can be used to generate thread dumps from a command line or using a shell script.

After you generate a series of thread dumps, Oracle recommends that you run the results through the ThreadLogic tool. This tool uses analysis algorithms and recommends solutions, or advisories, for common problems, including for WebLogic applications. This tool builds on the Thread Dump Analyzer (TDA) by adding logic for common patterns found in application servers. ThreadLogic is able to parse Sun, JRockit, and IBM thread dumps and provide advice based on predefined and externally defined patterns.

Note:

Use the following method in Linux and UNIX to produce a thread dump:

kill  -3 pid

where pid is the ID of the OSM WebLogic Server processor.

Monitoring the Database

When you monitor the operating system on the database server, make sure the system has enough idle CPU and Input/Output (I/O) wait times are low (nothing more than few percent of I/O waits are acceptable).

You can monitor the database by using Oracle Enterprise Manager, a web-based tool that allows you to manage Oracle software. For more information about Enterprise Manager, see Oracle Enterprise Manager Cloud Control documentation.

Updating Database Schema Statistics

For production databases, Oracle recommends that you update your database schema statistics regularly during an off-peak time of day.

To update the database schema statistics:

Log into SQL*Plus as the OSM database user, using the primary database schema.
Determine the ESTIMATE_PERCENT parameter you would like to use for statistics gathering.

Oracle recommends setting the ESTIMATE_PERCENT parameter of GATHER_SCHEMA_STATS to DBMS_STATS.AUTO_SAMPLE_SIZE to maximize performance gains while achieving necessary statistical accuracy. With a larger ESTIMATE_PERCENT value, statistics gathering for a large database could take several hours. If you prefer a faster, less accurate result, use a small ESTIMATE_PERCENT value such as 1.

Run the following procedure, substituting the ESTIMATE_PERCENT value that you determined for percent.

BEGIN
DBMS_STATS.GATHER_SCHEMA_STATS (
OWNNAME=>USER, 
ESTIMATE_PERCENT=>percent,
GRANULARITY =>'ALL',
CASCADE =>TRUE, 
BLOCK_SAMPLE=>TRUE);
END;

Managing Log Files

Oracle Diagnostic Logging (ODL) is the system logging service that is used by most Oracle Fusion Middleware applications. The system writes diagnostic messages to log files. Each message includes such information as the time, component ID, and user.

You can manage ODL log files using Oracle Enterprise Manager, the WebLogic Server Administration Console, or WebLogic Scripting Tool (WLST) commands. For more information about understanding ODL messages and log files, see Oracle Fusion Middleware Administrator's Guide.

Managing Log Files Using Enterprise Manager Fusion Middleware Control

Using Enterprise Manager, you can view and search log files, and configure settings for log files. For more information about managing log files and diagnostic data, see Oracle Fusion Middleware Administrator's Guide.

View and manage log files using the Enterprise Manager Fusion Middleware Control plug-in. For information about enabling the Enterprise Manager template during OSM installation, see the topic about creating the WebLogic server domain in OSM Installation Guide.

To access Enterprise Manager, enter the following URL into your browser:

http://host:port/em

where host is the IP address of the computer on which the WebLogic administration server is installed, and port is the port for the administration server domain.

Viewing Log Files and Messages Using Fusion Middleware Control

You can view log files and messages, as well as search and filter log information, by using the Log Messages page. For information about searching and filtering log information, see the topic about viewing log files and their messages in Oracle Fusion Middleware Administrator's Guide.

To open the Log Messages page:

From the navigation pane, expand WebLogic Domain, and then expand the domain.
Right-click the server on which you want to view log files and messages, and select Logs, and then View Log Messages.

The Log Messages page is displayed.

Configuring Logs Using Fusion Middleware Control

You can change the log settings of managed servers and Java components using Fusion Middleware Control. For information, see the topic about configuring settings for log files in Oracle Fusion Middleware Administrator's Guide.

Note:

You must apply log configuration settings to each server that you want to view logs for. Fusion Middleware Control allows you to view logs and log messages for each server in your OSM deployment.

From the Log Configuration page, you can configure the following:

Log Levels: Allows you to configure the log level for persistent and runtime loggers. For more information, see "About Log Severity Levels."
Log Files: Allows you to create and view log handlers, or edit, view, and delete the configuration of a log handler.
QuickTrace: Allows you to create and view QuickTrace handlers, or edit, view, and delete the configuration of a QuickTrace handler. Using QuickTrace, you can trace messages from specific loggers and store the messages in memory.

To open the Log Configuration page:

From the navigation pane, expand WebLogic Domain, and then expand the domain.
Right-click the server where you want to create and edit log file configuration, and select Logs, and then Log Configuration.

The Log Configuration page is displayed.

Managing Log Files Using WebLogic Server Administration Console

Use the Administration Console to view log files, configure the log view, and change log size and rotation. For more information about configuring logging, see WebLogic Server Administration Console online Help.

Viewing Log Files

You can view the OSM log files using WebLogic Server Administration Console.

To view the most recent messages in the log files:

Log in to the WebLogic Server Administration Console.
In the left pane of the Console, expand Diagnostics and select Log Files.
In the Log Files table, select the radio button next to the name of the log you want to view, and then click View.

The page displays the latest contents of the log file, up to 500 messages in reverse chronological order, meaning the messages at the top of the window are the most recent messages that the server has generated.

The log viewer does not display messages that have been rotated into archive log files.

Configuring the Log View

OSM logs everything that happens in the system, which can make the logs file quite large. Using the WebLogic Server Administration Console, you can configure the log views to show only the most recent messages.

To configure the view of the logs:

Log in to the WebLogic Server Administration Console.
From the log files page, click Customize this table.
Choose the filter and view options that you want, and then click Apply. To add an option to the Chosen column, highlight it in the Available column, and then click the arrow that points to the Chosen column. To deselect the option, reverse this action.

Log Size and Rotation

It is important to manage the log file's maximum size and rotations to prevent the logs from consuming disk resources. Rotation refers to log files that are rotated into archived log files.

To manage log settings:

Log in to the WebLogic Server Administration Console.
Click Environment/Servers/server.

The General Configuration page for the selected server is displayed.
Click the Logging tab.

A page that displays the logging settings for the selected server is displayed.
Modify the Rotation file size to 25,000 kilobytes and be sure to limit the number of Files to retain. Modify these settings for both tabs, General and HTTP.

About Log Severity Levels

To control the volume of log messages generated and written to the ODL log files, you assign severity levels to the various areas of the application that generate their own discrete messages. Some diagnostic message types have levels within them that you can set for finer logging granularity. If you do not assign a severity level, the severity level is NOTIFICATION:1.

The following is an descending list of the severity levels, starting with the most severe:

ERROR: (Most severe) A serious problem that requires immediate attention from the administrator and is not caused by a bug in the product
WARNING: A potential problem that should be reviewed by the administrator
NOTIFICATION: A major lifecycle event, such as the activation or deactivation of a primary sub-component or feature. NOTIFICATION:1 is the default level. NOTIFICATION:16 provides a finer level of granularity for reporting normal events.
TRACE: (Least severe) Trace or debug information for events that are meaningful to administrators, such as public API entry or exit points. For example, TRACE:16 provides detailed trace or debug information that can help diagnose problems with a particular subsystem.

If an event occurs inside a given category that triggers a message below the severity level assigned to the category (that is, less severe than the assigned level), ODL does not generate the message.

Configuring Log Levels Using the logging.xml File

You can change logging level configuration by modifying the logging.xml file. You can modify both log handlers and loggers. For more information about working with the logging.xml file, see the topic about log handler and logger configuration in Fusion Middleware Administrator's Guide for Oracle Identity Manager.

The logging.xml file has the following basic structure:

<logging configuration>
  <log_handlers>
    <log_handler name='console-handler' level="NOTIFICATION:16"></log_handler>
    <log_handler name='odl-handler'></log_handler>
    <!--Additional log_handler elements defined here....-->
  </log_handlers>
  <loggers>
    <logger name="example.logger.one" level="NOTIFICATION:16">
      <handler name="console-handler"/>
    </logger>
    <logger name="example.logger.two" />
    <logger name="example.logger.three" />
    <!--Additional logger elements defined here....-->
  </loggers>
</logging_configuration>

To change log levels using the logging.xml file:

Open the logging.xml file, which is in the following location:

domain_home/config/fmwconfig/servers/server name/logging.xml
Add or change the log level for the appropriate log handler.

For example, change:
```
<log_handler name='odl-handler' class='oracle.core.ojdl.logging.ODLHandlerFactory' filter='oracle.dfw.incident.IncidentDetectionLogFilter'>
```
To the following:
```
<log_handler name='odl-handler' class='oracle.core.ojdl.logging.ODLHandlerFactory' level='TRACE:32' filter='oracle.dfw.incident.IncidentDetectionLogFilter'>
```
This log handler is now able to log debug messages.

Note:

Even though the logging.xml file is configured to allow the handler to log debug messages, you must also enable the corresponding OSM Java class to the correct logging level.

Managing Logs Using WLST

You can view and query server logs by using WebLogic Scripting Tool (WLST) commands, as well as changing log levels. You can list the log files for an Oracle WebLogic Server domain, server, an Oracle instance, or component using the WLST listLogs command. You can search the log files by using the WLST displayLogs command and narrow the search by specifying criteria, such as time, component ID, or message type. For example:

displayLogs(target='opmn:asinst_1/ohs1', last=5)

Logs are automatically aggregated across all managed servers in a cluster. Log viewing commands work whether you are connected or not connected to a WebLogic Server.

To run WLST in interactive mode:

Do one of the following:
- For UNIX and Solaris platforms, enter the following command:
```
MW_home/Middleware/oracle_common/common/bin/wlst.sh
```
- For Windows platforms, enter the following command:
```
MW_home/Middleware/oracle_common/common/bin/wlst.cmd
```

Note:

Most of the WLST logging commands require that you are running in the domainRuntime tree. For example, to connect and to run in the domainRuntime tree, use the following commands:

connect('WLS_username', 'WLS_password', 'WLS_hostname:WLS_port_number')
domainRuntime()

For more information about managing ODL files by using WLST, see the topic about managing log files and diagnostic data in Oracle Fusion Middleware Administrator's Guide.

Configuring Log Levels Using WLST

To configure log levels using a WLST command:

Enter WLST in interactive mode.
Run the setLogLevel command. For example:
```
setLogLevel(target='soa_server1', logger='oracle.soa', level='WARNING')
```
In this example, the soa_server1 server logging level is set to WARNING. For more information, see "About Log Severity Levels."

For more information about managing ODL files by using WLST, see the topic about managing log files and diagnostic data in Oracle Fusion Middleware Administrator's Guide.

Secure vs Non-Secure Log Filtering

A stack trace is a list of the method calls that an application is in the middle of at the time an exception is thrown. Running OSM with stack trace logging enabled can be important for debugging an application during runtime.

Using the oms-config.xml parameter, enable_log_stacktraces, you can enable or disable logging stack traces. By default, this parameter is enabled. If there is a security concern about having log stack traces enabled, you can disable this parameter. For more information about setting this parameter, see "Configuring OSM with oms-config.xml".

Managing OSM Metrics

OSM provides a sample Grafana dashboard that can be used to visualize OSM metrics available from a Prometheus data source. OSM relies on Prometheus to scrape and expose these metrics.

See the following topics for further details:

Configuring Prometheus for OSM Metrics

Configure the scrape job in Prometheus for OSM as follows:

- job_name: 'job_name'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'
    metrics_path: /OrderManagement/metrics
    static_configs:
    - targets:
      - MS1_hostname:MS1_port
      - MS2_hostname:MS2_port
      - MSn_hostname:Msn_port

where:

job_name refers to a particular job. For example, COM_Production, SOM_UAT, and so on. Use this to distinguish the various OSM instances such as COM_Production, COM_Pre-prod, SOM_QA, SOM_UAT, and so on. Each of these instances will have its own job in the scrape configuration.
MS1 refers to managed server 1, MS2 refers to managed server 2 and so on.

Viewing OSM Metrics Without Using Prometheus

The OSM metrics can be viewed at:

http://hostname:port/OrderManagement/metrics

This only provides metrics of the managed server that is serving the request. It does not provide the consolidated metrics for the entire cluster. Only Prometheus Query and Grafana dashboards can provide the consolidated metrics.

Viewing OSM Metrics in Grafana

OSM metrics scraped by Prometheus can be made available for further processing and visualization.

Exposed Order Metrics

The following OSM metrics are exposed via Prometheus APIs.

Note:

All metrics are per managed server. Prometheus Query Language can be used to combine or aggregate metrics across all managed servers.
All metric values are short-lived and indicate the number of orders (or tasks) in a particular state since the managed server was last restarted.
When a managed server restarts, all the metrics are reset to 0. These metrics do not refer to the exact values, which can be queried via OSM APIs such as Web Services and XML API.

Order Metrics

The following table lists order metrics exposed via Prometheus APIs.

Table 10-1 Order Metrics Exposed via Prometheus APIs

Name	Type	Help Text	Notes
osm_orders_created	Counter	Counter for the number of Orders Created	N/A
osm_orders_completed	Counter	Counter for the number of Orders Completed	N/A
osm_orders_failed	Counter	Counter for the number of Orders Failed	N/A
osm_orders_cancelled	Counter	Counter for the number of Orders Cancelled	N/A
osm_orders_aborted	Counter	Counter for the number of Orders Aborted	N/A
osm_orders_in_progress	Gauge	Gauge for the number of orders currently in the In Progress state	N/A
osm_orders_amending	Gauge	Gauge for the number of orders currently in the Amending state	N/A
osm_short_lived_orders	Histogram	Histogram that tracks the duration of all orders in seconds with buckets for 1 second, 3 seconds, 5 seconds, 10 seconds, 1 minute, 3 minutes, 5 minutes, and 15 minutes. Enables focus on short-lived orders.	Buckets for 1 second, 3 seconds, 5 seconds, 10 seconds, 1 minute, 3 minutes, 5 minutes, and 15 minutes.
osm_medium_lived_orders	Histogram	Histogram that tracks the duration of all orders in minutes with buckets for 5 minutes, 15 minutes, 1 hour, 12 hours, 1 day, 3 days, 1 week, and 2 weeks. Enables focus on medium-lived orders.	Buckets for 5 minutes, 15 minutes, 1 hour, 12 hours, 1 day, 3 days, 7 days, and 14 days.
osm_long_lived_orders	Histogram	Histogram that tracks the duration of all orders in days with buckets for 1 week, 2 weeks, 1 month, 2 months, 3 months, 6 months, 1 year and 2 years. Enables focus on long-lived orders.	Buckets for 7 days, 14 days, 30 days, 60 days, 90 days, 180 days, 365 days, and 730 days.
osm_order_cache_entries_total	Gauge	Gauge for the number of entries in the cache of type order, orchestration, historical order, closed order, and redo order.	N/A
osm_order_cache_max_entries_total	Gauge	Gauge for the number of maximum entries in the cache of type order,orchestration, historical order, closed order, and redo order.	N/A

Labels For All Order Metrics

The following table lists labels for all order metrics.

Table 10-2 Labels for All Order Metrics

Label Name	Sample Value	Notes	Source of the Label
cartridge_name_version	view_framework_demo_1.0.0.0.0	Combined Cartridge Name and Version	OSM Metric Label Name/Value
order_type	vf_demo_web	OSM Order Type	OSM Metric Label Name/Value
server_name	ms1	Name of the Managed Server	OSM Metric Label Name/Value
instance	10.244.0.198:8081	Indicates the IP address and port of the host from which this metric is being scraped.	Prometheus Service Discovery
job	COM_Production	Job name in Prometheus configuration which scraped this metric.	Prometheus Service Discovery

Task Metrics

The following metrics are captured for Manual or Automated Task Types only. All other Task Types are currently not being captured.

Table 10-3 Task Metrics Captured for Manual or Automated Task Types Only

Name	Type	Help Text
osm_tasks_created	Counter	Counter for the number of Tasks Created
osm_tasks_completed	Counter	Counter for the number of Tasks Completed

Labels for all Task Metrics

A task metric has all the labels that an order metric has. In addition, a task metric has two more labels.

Table 10-4 Labels for All Task Metrics

Label	Sample Value	Notes	Source of Label
task_name	enter_account_information	Task Name	OSM Metric Label Name/Value
task_type	A	A for Automated M for Manual	OSM Metric Label Name/Value

Label

Sample Value

Notes

Source of Label

task_name

enter_account_information

Task Name

OSM Metric Label Name/Value

task_type

A for Automated

M for Manual

OSM Metric Label Name/Value

Managing WebLogic Server Metrics

OSM provides a sample Grafana dashboard that you can use to visualize WebLogic server metrics available from a Prometheus data source. You use WebLogic Monitoring Exporter (WME) to expose the WebLogic server metrics. WebLogic Monitoring Exporter is part of the WebLogic Kubernetes toolkit. It is an open source project, based at: https://github.com/oracle/weblogic-monitoring-exporter. While the metrics are available via WME Restful Management API endpoints, OSM relies on Prometheus to scrape and expose these metrics. This version of OSM supports WME 1.3.0. See the WME documentation for details on configuration and the exposed metrics.

The following topics describe a sample integration:

Deploying WebLogic Monitoring Exporter (WME) in OSM

To deploy WebLogic Monitoring Exporter:

Generate the WME WAR file by running the following command:


mkdir -p ~/wme
cd ~/wme
 
curl -x $http_proxy -L https://github.com/oracle/weblogic-monitoring-exporter/releases/download/v1.3.0/wls-exporter.war -o wls-exporter.war
curl -x $http_proxy https://raw.githubusercontent.com/oracle/weblogic-monitoring-exporter/v1.3.0/samples/kubernetes/end2end/dashboard/exporter-config.yaml -o exporter-config.yaml
 
jar -uvf wls-exporter.war exporter-config.yaml

This command updates the wls-exporter.war file with the exporter-config.yaml configuration file.

Deploy the WME WAR file by running the following command:

java -cp path_to_weblogic_server_lib weblogic.Deployer -adminurl t3://host_name:adminserver_port -user wls_admin_username -password wls_admin_password -deploy -name name_of_the WME_WAR_file -source  path_to MWE_WAR_file -targets wls_server_targets_list
 
### Example :
java -cp /../../Oracle/WLS/12_2_1_4/wlserver/server/lib/weblogic.jar weblogic.Deployer -adminurl t3://localhost:7001 -user weblogic -password password -deploy -name wls-exporter -source  /../../wme/wls-exporter.war -targets AdminServer,Cluster_Name

Configuring the Prometheus Scrape Job for WME Metrics

Configure the scrape job in Prometheus as follows:

- job_name: 'wme_job_name'
  metrics_path: wls-exporter/metrics
  basic_auth:
    username: weblogic_username
    password: weblogic_password
  static_configs:
  - targets: [AdminServer_hostname:AdminServer_port]
    labels:
      # The namespace label enables multiple related instances to be grouped.
      # For a given WebLogic server, the namespace used in a WME Prometheus job must match
      # the namespace used for the corresponding OSM Prometheus job.
      # In a sample Grafana dashboard, the specified namespace is displayed under the Project
      # drop-down menu.
      namespace: namespace
      # The weblogic_domainUID label uniquely identifies an OSM instance within a given namespace.
      # For a given WebLogic server, the weblogic_domainUID used in a WME Prometheus job must match
      # the weblogic_domainUID used for the corresponding OSM Prometheus job. 
      # In a sample Grafana dashboard, the specified weblogic_domainUID is displayed under the
      # Instance drop-down menu.
      weblogic_domainUID: weblogic_domainUID
      # The weblogic_serverName label must match the server name configured in WebLogic.
      weblogic_serverName: AdminServer
 
  # Repeat this pattern for each managed server
  - targets: [MSn_hostname:Msn_port] 
    labels:
      namespace: namespace
      weblogic_domainUID: weblogic_domainUID
      weblogic_serverName: MSn

The namespace label enables multiple related instances to be grouped. For a given WebLogic server, the namespace used in a WME Prometheus job must match the namespace used for the corresponding OSM Prometheus job. In a sample Grafana dashboard, the specified namespace is displayed under the Project drop-down menu.

The weblogic_domainUID label uniquely identifies an OSM instance within a given namespace. For a given WebLogic server, the weblogic_domainUID label used in a WME Prometheus job must match the weblogic_domainUID used for the corresponding OSM Prometheus job. In a sample Grafana dashboard, the specified weblogic_domainUID is displayed under the Instance drop-down menu.

The weblogic_serverName label must match the server name configured in WebLogic.

To enable correlation with WME metrics, the namespace and weblogic_domainUID labels have been added to the corresponding scrape job definition for OSM metrics. Note that, with these new labels, the dashboard's JSON files can be reused between OSM traditional and OSM cloud native deployments.

- job_name: 'osm_job_name'
  # metrics_path defaults to '/metrics'
  # scheme defaults to 'http'
  metrics_path: OrderManagement/metrics
  static_configs:    
  - targets: [MSn_hostname:Msn_port]
    labels:
      namespace: namespace
      weblogic_domainUID: weblogic_domainUID

Viewing WebLogic Monitoring Exporter (WME) Metrics Without Using Prometheus

To view WME metrics of the admin server without using Prometheus, access the following URL:

http://adminserver_host:adminserver_port/wls-exporter/metrics

To view WME metrics of managed servers without using Prometheus, access the following URL:

http://managedServerN_host:managedServerN_port/wls-exporter/metrics

Viewing WebLogic Monitoring Exporter Metrics in Grafana

OSM provides sample Grafana dashboards to get you started with visualizations. The sample OSM and WebLogic by Server dashboard provides a combined view of OSM cloud native and WebLogic Monitoring Exporter metrics for one or more managed servers for a given instance in the selected project namespace.

Import the dashboard JSON file from OSM_SDK/samples/Grafana into your Grafana environment, selecting Prometheus as the data source.

Managing Database Connections

In OSM, database connections are managed through database pools that are set up in WebLogic Server.

The database pool connections can be configured through the WebLogic Server Administration Console by clicking Services/JDBC/Data Sources/oms_pool/Connection Pool.

Use the parameters on the Connection Pool page to modify and tune your connection pool settings.

Using JMS Queues to Send Messages

OSM uses JMS Queues and Topics, which are both JMS Destinations. Queues follow a point-to-point communication paradigm, while Topics follow the publish and subscribe paradigm.

Note:

In an OSM clustered environment, you must use JMS queues as a JMS destination to receive JMS events. Do not use JMS topics in an OSM clustered environment.

When OSM sends data to an external system, such as UIM or ASAP, it does so by sending JMS messages to the appropriate JMS request queue of an external system.

If the external system is not processing the requests from OSM, the queues get backlogged. It is important to be able to monitor the size of the JMS queues in order to know whether or not they are backing up.

To monitor the JMS queues:

Login to the WebLogic Administration Console

Click Services/Messaging/JMS Servers/oms_jms_server.

The General Configuration page is displayed.
Click the Monitoring tab and then click Active Destinations.

A list of active destinations targeted to the server is displayed.

Note:

The default view of the table does not contain the Consumers Current column. We recommend that you customize the table using the Customize link to include this column, along with any other customizations you may want to make.

The Consumers Column column defines the current number of listeners on the destination. If a destination does not have any listeners, the external system does not receive the messages.

The Messages Current column defines the current number of unprocessed messages in the JMS destination. A large number of messages (for example, 10,000) in this destination is a problem. It means that the system is not keeping up, that the messages are not getting processed, or that the messages are getting processed but errors are occurring and the messages are getting put back on the destination.

OSM has the following JMS destinations present:

oms_behavior_queue: Used for customizing task assignment
oms_events: Internal destination used for events such as automation, notifications, and task state changes
oms_order_events: Used for order state changes such as OrderCreateEvent, OrderStateChangeEvent, AmendmentStartedEvent, OrderCancelledEvent
oms_order_updates: Internal destination used for processing amendments
oms_signal_topic: Internal destination used to trigger a metadata refresh

Monitoring the Event Queue

The destination oms_events is the JMS destination to which OSM events are sent. OSM events are sent when tasks change states, or when notifications occur.

The number of consumers for the oms_events is determined by which plug-ins are configured. If plug-ins are configured, the number of consumers must not be 0.

If there is a problem with automation plug-ins getting invoked, check the consumers queue and the messages queue.

If the consumers queue is less than the number of plug-ins, the plug-ins are not configured correctly. Check the OSM_home\SDK\Samples\DatabasePlugin\map\ automationmap.xml file and make sure that all of the plug-ins are deployed.

If the messages queue keeps getting larger, the plug-ins may not be committing the transactions during processing of the events. Verify the plug-in code and check the log files.

Sending Data to External Systems Using Plug-Ins

If there are external systems deployed to the same WebLogic instance as OSM, when you monitor the JMS destinations, watch for the following.

Note:

The important columns are Consumers, Messages, and Messages Received.

If the number in the messages column for these queues continues to grow, the external system may not be processing the messages sent by OSM. You must check to see if the external system is working properly.

If the number of consumers for the queues is 0, the external system may not have configured its listeners properly. Check to see if the external system is configured properly.

About OSM and XA Support

The OSM database does not support XA transactions because the Oracle thin-client driver used for JDBC connections does not support XA. However, the OSM WebLogic Server configuration uses an XA emulation feature in order to get a two-phase commit across JMS/JDBC automation transactions.

Even though OSM uses a non-XA driver for database transactions, external XA resources can still participate in transactions. For example, JMS bridges can be XA-enabled for an outside application, but the OSM side of the transaction still uses the non-XA emulated two-phase commit. Note that this also applies to JMS queues that support Application Integration Architecture (AIA) cartridges.

Using Work Managers to Prioritize Work

You use WebLogic Administration Console to configure work managers, which prioritize OSM work and manage threads. WebLogic Server uses a single thread pool in which all types of work are executed. Work managers allow you to define rules and run-time metrics that WebLogic Server uses to prioritize this work.

For more information about using work managers to optimize scheduled work, see Oracle Fusion Middleware Configuring Server Environments for Oracle WebLogic Server.

Creating and Configuring Work Managers

When you run the installer and restart the WebLogic Server, the system creates and configures default work managers and components.

Table 10-5 shows the default work managers and components that are created and configured by the installer.

Table 10-5 Default Work Managers and Components

Work Manager or Component	Configuration Properties	Description
Maximum Threads Constraint	Name: osmMaxThreadConstraint Count: 90% of the number of connections in the OSM datasource	Used by all OSM Work Managers to prioritize work.
Minimum Threads Constraint	Name: osmMinThreadConstraint Count: 10	Used by the work managers osmTaskClientWorkManager and osmOmClientWorkManager to guarantee that some threads are reserved for the web clients.
Work Manager	Name: osmAutomationWorkManager Maximum Threads Constraint: osmMaxThreadConstraint	Used to process work performed by automation tasks.
Work Manager	Name: osmXmlWorkManager Maximum Threads Constraint: osmMaxThreadConstraint	Used to process requests coming from external clients for the OSM XML API.
Work Manager	Name: osmTaskClientWorkManager Maximum Threads Constraint: osmMaxThreadConstraint Minimum Threads Constraint: osmMinThreadConstraint Request Class: osmFairShareReqClass	Used to process requests from manual users using the Task web client.
Work Manager	Name: osmWsJmsWorkManager Maximum Threads Constraint: osmMaxThreadConstraint	Used to process OSM JMS Web Service requests.
Work Manager	Name: osmWsHttpWorkManager Maximum Threads Constraint: osmMaxThreadConstraint	Used to process OSM HTTP Web Service requests.
Work Manager	Name: osmOmClientWorkManager Maximum Threads Constraint: osmMaxThreadConstraint Minimum Threads Constraint: osmMinThreadConstraint	Used to process requests from manual users using the Order Management web client.

In addition to the defaults, you can create and configure other work managers and components. For more information, see Oracle Fusion Middleware Configuring Server Environments for Oracle WebLogic Server.

To create a new work manager:

Log in to the WebLogic Server Administration Console.
In the left pane of the Console, expand Environment and select Work Managers.

This page displays a summary of the work managers configured for the system.
Click New.

The Create a New Work Manager Component page displays.
Select Work Manager, and then click Next.
Enter a name for the new Work Manager, and then click Next.
Select a deployment target from the list of available targets.
Click Finish.

The new work manager is displayed in the summary of work managers list.

Overriding the Internet Explorer Language in the OMS Web Clients

If the Internet Explorer installation that is used to access the OMS web clients is set to a language other than English, and this language matches one of the properties files included in the oms.ear file, the web client prompts are displayed in the non-English language.

The language used in the web clients is controlled by the resources.properties file. Additional language property files in the oms.ear file include:

resources_cs.properties (Czech language properties file)
resources_zh.properties (Chinese language properties file)

To remove support for a non-English language, unpack the oms.ear file, remove the corresponding properties file, repack the oms.ear file. For more information about working with oms.ear file, see OSM Developer's Guide.

For example, if the browser is set to use the Czech language, and the resources_cs.properties file exists in the oms.ear file, the web client prompts are displayed in Czech. Removing the resources_cs.properties file causes the web client prompts to be displayed in English, even though the browser language setting is still configured to the Czech language.

About Metrics Data

The OSM Order Metrics Manager measures and reports managed server and domain-wide performance metrics, and traces performance for your OSM environment. You view metric data using an interface called Oracle Dynamic Monitoring Service (DMS). For more information about DMS, see Oracle Fusion Middleware Performance and Tuning Guide.

Viewing Metrics Data

Metrics data is gathered and displayed using tables. You view OSM-specific metric data, which is displayed in the DMS Metrics tables and Aggregated Metrics tables. You can also use the DMS interface to view non-OSM metric data, for example, WebLogic and non-J2EE metrics.

You can also view metrics data by using Oracle Enterprise Manager. For information about viewing metrics using Enterprise Manager, see Oracle Fusion Middleware Performance and Tuning Guide. For more information about viewing and analyzing metric data using Oracle Application Management Pack (AMP) for Enterprise Manager, see "Analyzing Metric Data".

For information about installing metrics rules (ADML) files, see OSM Installation Guide.

Figure 10-1 shows the DMS user interface for an environment that has Oracle HTTP server configured. A list of links to metrics tables is on the left-hand side. When you click the link to a metrics table, the table opens in the pane on the right.

Figure 10-1 DMS Order Requests Metrics Table

Description of "Figure 10-1 DMS Order Requests Metrics Table"

To view metrics tables using DMS:

In a browser, enter one of the following URLs:
```
http://hostname:port/dms

https://hostname.dnsdomainname.tld:sslport/dms
```
where hostname is the name of the WebLogic administration server, port is the port for the administration server domain, and sslport is the secure port for the administration server domain.

The DMS Spy login screen is displayed.
Log in using your WebLogic username and password.

A list of all metric table names is displayed.
Click the metric table name for the table that you want to view.

The details of the metric table are displayed in the right-hand pane.

Note:

For any table, you can display a list of the metric column names, a description of each column name, and the derivatives and units for the metric. To view this information, click the Metric Definition link at the bottom of the table.

About DMS Metrics Tables

The DMS metrics tables display OSM metrics. You can view these metrics using the DSM Spy servlet, or using other systems, such as Enterprise Manager using MBeans.

DMS metric tables are named using the following convention: industry_application_MetricName. For example, Comms_OrderManagement_OrdersRuntime.

About WebLogic Metrics Tables

Using the links to WebLogic Metrics tables, you can view data about the WebLogic server and JMX. The tables include information about management of Java memory and WebLogic diagnostics, configuration, and runtime data.

About Non-J2EE Metrics Tables

The data for these metrics tables comes from an Oracle HTTP server. The tables include data about the PL/SQL API, the Oracle HTTP server, and the Oracle Process Manager and Notification Server (OPMN).

If you do not see a link to these tables, your domain is not connected to an Oracle HTTP server for load balancing. For information about adding and configuring an Oracle HTTP server, see the topic about configuring the WebLogic server domain for an OSM cluster in OSM Installation Guide.

About Aggregated Metrics Tables

Because OSM aggregated metrics are based on existing metric tables, aggregation allows you to specify the new aggregated metric tables and the steps by which to compute their data. The rules are written in XML configuration files, which are also called metric rules files or ADML files. For more information about updating metric rules, see the topic about manually loading metric rules files in OSM Installation Guide.

Aggregated metric tables are named using the following convention: industry_application:MetricName. For example, Comms_OrderManagement:Cluster_all_orders.

About Metric Rules Files

Metric rules files, or ADML files, contain the aggregate metric configurations. All aggregate metrics are defined in these files. For OSM, there are the following aggregate files:

Server based: for managed server based aggregation. The file name is prefixed with server-, which indicates the metric rules contained in the XML file are for aggregating OSM metrics across the managed server. The full file name is: server-oracle-comms_osm-11.0.xml.
Domain wide: for aggregating OSM metrics across the domain. The file name is prefixed with domain-. The full file name is: domain-oracle-comms_osm-11.0.xml.

Analyzing Metric Data

You can view the metric data that is gathered by the system about your environment by accessing the DMS interface that is provided by default with your OSM installation. The tables provide information about various aspects of your OSM environment, and you can compare the pieces of data over time to determine whether there is a problem. For example, if you want to make sure the system is properly reallocating resources after cluster resizing, you view the related aggregated metric data in the WebLogic cluster tables.

You can also view that same data using the Application Management Pack, which is a plug-in that is available in Enterprise Manager. This view provides a graphical representation of metric data, which allows you to analyze and interpret data over time. For information about viewing metrics using Enterprise Manager, see Oracle Fusion Middleware Performance and Tuning Guide. For information about using Application Management Pack, see Oracle Application Management Pack for Oracle Communications System Administrator's Guide.

Figure 10-2 shows the Application Management Pack dashboard for an OSM system. The interface also displays metrics by server, order type, and cartridge.

Figure 10-2 OSM System Data in the AMP Dashboard

Description of "Figure 10-2 OSM System Data in the AMP Dashboard"

Discovering a Cluster Database Target

You can use Enterprise Manager to associate a Oracle RAC database to an OSM target, and then view the association in the Topology page.

To associate a Oracle RAC database to an OSM target, you must first discover the Oracle RAC database as a cluster database target. Discovering the Oracle RAC database as a cluster database target is a two-step process of adding a cluster target, and then discovering the cluster database target by selecting the one that you added.

For more information about discovering targets, see Enterprise Manager Cloud Control Administrator's Guide.

To add a cluster target:

In Enterprise Manager, click Setup, Add Target, and then click Add Target Manually.
Select Add Targets Using Guided Process.
In the Target Types list, select Oracle Cluster and High Availability Service.
Click Add Using Guided Process, and discover the Oracle RAC database.
Save the Oracle RAC database as a cluster target.

The Oracle RAC database is added as a cluster target.

To discover and associate the cluster database target:

In Enterprise Manager, click Setup, Add Target, and then click Add Target Manually.
Select Add Targets Using Guided Process.
In the Target Types list, select Oracle Database, Listener, and Automatic Storage Management.
Click Add Using Guided Process, and select the added cluster target.
Follow the steps in the wizard to add it as a cluster database target.

The Oracle RAC database is added as a cluster database target.
On the dashboard page of the OSM target, in Enterprise Manager, click Associate RAC Database.

Note:

To successfully associate the Oracle RAC database to the cluster target, ensure that the OSM target servers are running.

The Oracle RAC database is associated to the target that you specified.