6 Troubleshooting Offline Mediation Controller

Learn how to troubleshoot problems in Oracle Communications Offline Mediation Controller.

Topics in this document:

Troubleshooting Checklist

When any problems occur, it is best to do some troubleshooting before you contact Oracle Support:

  • You know your installation better than Oracle Support does. You know if anything in the system has been changed, so you are more likely to know where to look first.

  • Troubleshooting skills are essential. Relying on Oracle Support to research and solve all of your problems prevents you from being in complete control of your system.

If you have a problem with your Offline Mediation Controller system, ask yourself these questions first, because Oracle Support will ask them of you:

  • What exactly is the problem? Can you isolate it?

    Oracle Support requires a clear and concise description of the problem, including the date it began to occur.

  • What do the log files say?

    This is the first thing that Oracle Support asks for. Check the error log for the Offline Mediation Controller component that is causing the issue.

  • Has anything changed in the system? Did you install any new hardware or new software? Did the network change in any way? Does the problem resemble another one you had previously? Has your system usage recently jumped significantly?

  • Is the system otherwise operating normally? Has the response time or the level of system resources changed? Are users complaining about additional or different problems?

Using Error Logs to Troubleshoot Offline Mediation Controller

Offline Mediation Controller error log files provide detailed information about system problems. If you are having a problem with Offline Mediation Controller, look in the log files.

Offline Mediation Controller logs specific details about actions performed in the Offline Mediation Controller GUI in Administration Server log files.

Getting Help for Offline Mediation Controller Problems

If you are unable to resolve the Offline Mediation Controller issue, contact Oracle Support.

Before contacting Oracle Support, try to resolve the problem using the information logged in the log files. If this does not help to resolve the issue, note the following information:

  • A clear and concise description of the problem, including when it began to occur.

  • Relevant portions of the relevant log files.

  • Relevant configuration files.

  • Recent changes in your system after which the problem occurred, even if you do not think they are relevant.

  • List of all Offline Mediation Controller components and patches installed on your system.

When you are ready, report the problem to Oracle Support.

Common Problems and Their Solutions

Although Offline Mediation Controller is designed to be trouble-free, you might encounter a problem. You can view descriptions of some common problems along with their solutions:

Problem: ps: illegal option

When you start Node Manager or the Administrative Server, you receive a "ps: illegal option" error message.

Possible Cause

The ps script from the ucb package is used. You must use the ps script in the /usr/bin or /bin directory.

Solution

In the PATH environment variable, make sure that /usr/bin and /bin appear before /usr/ucb.

Problem: keytool error: java.io.FileNotFoundException While Creating Administration Server Certificate

After installing only Administration Server, when you run the createAdminSvrCert script to create the Administration Server certificate, you receive the "keytool error: java.io.FileNotFoundException: OMC_home/config/GUI/adminClientTruststore.jks (No such file or directory)" error message, where OMC_home is the directory in which Offline Mediation Controller is installed.

Possible Cause

The OMC_home/config/GUI/adminClientTruststore.jks file does not exist.

Solution

You must manually import the Administration Server's public certificate into the Administration Client's TrustStore.

To manually import the Administration Server's public certificate into the Administration Client's TrustStore:

  1. Log in to the system on which Offline Mediation Controller is installed.

  2. Run the following command:

    OMC_home/jre/bin/keytool -import -v -trustcacerts -alias adminServer -file OMC_home/config/adminserver/adminServer.cer -keystore OMC_home/config/GUI/adminClientTruststore.jks

    The Enter keystore password prompt appears.

  3. Enter the Administration Client's TrustStore password.

    The Trust this certificate prompt appears.

  4. Enter y, which trusts the certificate.

    The Administration Server's public certificate is imported into the Administration Server's TrustStore.

Problem: keytool error: java.io.FileNotFoundException While Creating Node Manager Certificate

After installing only Node Manager, when you run the createNodeMgrCert script to create the Node Manager certificate, you receive the "keytool error: java.io.FileNotFoundException: OMC_home/config/adminserver/adminServerTuststore.jks (No such file or directory)" error message.

Possible Cause

The OMC_home/config/adminserver/adminServerTuststore.jks file does not exist.

Solution

You must manually import the Node Manager's public certificate into the Administration Server's TrustStore.

To manually import the Node Manager's public certificate into the Administration Server's TrustStore:

  1. Log in to the system on which Offline Mediation Controller is installed.

  2. Run the following command:

    OMC_home/jre/bin/keytool -import -v -trustcacerts -alias nodeManager -file OMC_home/config/nodemgr/nodeManager.cer -keystore OMC_home/config/adminserver/adminServerTuststore.jks

    The Enter KeyStore password prompt appears.

  3. Enter the Administration Server's KeyStore (TrustStore) password.

    The Trust this certificate prompt appears.

  4. Enter y, which trusts the certificate.

    The Node Manager's public certificate is imported into the Administration Server's TrustStore.

Problem: com.maverick.ssh.SshException When FTP is Used in Collection or Distribution Cartridge Pack

When File Transfer Protocol (FTP) is used in the Collection or Distribution cartridge pack, you receive the "com.maverick.ssh.SshException" error.

Possible Cause

Not all required security providers are included in the Java security files.

Solution

Manually copy the following JAR files into the Java_home/jre/lib/ext directory (where Java_home is the directory in which the supported version of Java is installed) and then restart the system:

  • bcprov-ext-jdk15on-158.jar

  • bcprov-jdk15on-158.jar

To restart the system, see "Starting and Stopping Offline Mediation Controller".

Problem: Error While Starting Node Manager with SSL Enabled

When you start Node Manager with Secure Sockets Layer (SSL) enabled, you receive the following error:

Cannot support TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 with currently installed providers

Possible Cause

Not all required security providers are included in the Java security files.

Solution

Manually copy the JARs from the jce_policy-8.zip file into the Java_home/jre/lib directory and then restart the system. You can download this file from Oracle Technology Network:

https://www.oracle.com/java/technologies/javase-jce8-downloads.html

To restart the system, see "Starting and Stopping Offline Mediation Controller".

Problem: Memory Issues Caused by Duplicate Check Failures

While processing records in Offline Mediation Controller, you may encounter an issue where duplicate Call Detail Records (CDRs) are processed despite existing checks. This can lead to multiple processing of the same records, resulting in potential system overloads and the following error in the Node Manager log file:
2024-01-03 16:24:10;ERROR;Resources; Memory usage exceeds MAJOR threshold; 
Memory usage has exceeded 85% for 600 seconds.#EOR#

Possible Cause

Although there are mechanisms to check for duplicates, these mechanisms may fail under certain conditions, allowing duplicates to be processed again.

Solution

To fix the memory issues, follow these guidelines:
  • Fix the duplicate check failure: Ensure the Duplicate Check Enhancement Processor (EP) node is configured correctly. Also, ensure the Keep Track of Processed Files option is enabled in your ASCII Collection Cartridge (CC).
  • Adjust the CDR retention limit: For instance, reduce it from 30 days to a more manageable duration, like 7 days, to help mitigate memory issues.
  • Increase JVM memory settings: Allocate more than 3 GB to both the admin server and the node manager to handle high volumes of data without exceeding memory thresholds.

Problem: Database Connection Issues with JDBC DC Node

When attempting to connect the JDBC Distribution Cartridge (DC) node to the Oracle database, you may encounter the following error messages:

  • javax.crypto.BadPaddingException: Given final block not properly padded.

  • ORA-01005: null password given; logon denied: Unable to obtain a connection from the factory.

Possible Cause

These issues often arise from discrepancies in wallet configurations between the Offline Mediation Controller Administration Client and your cloud native installation. Specifically, different wallets may be in use, or the wallet may not be properly mounted in your cloud native system for access.

Solution

To resolve the connectivity issues with the JDBC DC node:
  1. Ensure that both the Administration Client and cloud native installation reference the same wallet to avoid configuration discrepancies.

    • Administration Client: The wallet location is specified in the UDCEnvironment file in the OCOMC_WALLET_LOCATION parameter.

    • Offline Mediation Controller cloud native: The wallet location is specified in the values.yaml file’s walletFolder key.

  2. Confirm that the wallet is correctly mounted so that the Administration Client can access it. For more information, see "Post-Installation Tasks for Administration Client" in the Cloud Native Installation and Administration Guide.

  3. After making the configuration changes, either create a new node or save the existing database connection details again.

Problem: BRSOverloadException During CDR Processing

While processing large numbers of Call Detail Records (CDRs) in an Offline Mediation Controller production environment, you may encounter an issue where the CDRs get stuck in the Offline Mediation Controllers ECE DC node's input directory.

You may find the following error message in the Oracle Communications Elastic Charging Engine (ECE) DC log file: 

2024-09-11 16:59:12;ERROR;Connectivity; Error connecting to OCECE; BrsMessageBundle-
22017: BRS is currently overloaded, 11 pending counts waiting to be processed.;
oracle.communication.brm.charging.brs.BRSOverloadException: BrsMessageBundle-22017: BRS
is currently overloaded, 11 pending counts waiting to be processed.

Possible Cause

Offline Mediation Controller is processing a high volume of CDRs, leading to an overload situation in the ECE DC node.

Solution

To resolve this issue and prevent future occurrences:

  1. Enable Request Throttling in the ECE DC Node: You can manage incoming requests and prevent system overload by enabling request throttling on the ECE DC node. This feature works when the request queue exceeds a defined threshold (high-water mark), triggering the throttling mechanism. Once the load drops below a set threshold (low-water mark), regular processing resumes. For example, you can set the high-water mark to 1500 and the low-water mark to 500. Adjust these values based on your system's capacity and anticipated traffic. For more information about configuring these entries, see "Configuring OCECE DC Settings" in Offline Mediation Designer Help.

  2. Set the Oracle Communications Billing and Revenue Management (BRM) Pending Count: Increase the Pending Count value according to your requirements (for example, set it to 1500). This setting allows the system to handle a larger number of pending requests before triggering an overload exception. For more information, see “Configuring System Overload Protection” in BRM System Administrator's Guide.

Problem: Suspense Issue with Error Code 60015 (SYSTEM_ERR)

After a rolling restart of Elastic Charging Engine (ECE), you may encounter a suspense issue with error code 60015 (SYSTEM_ERR) and a Null Pointer Exception in the ECS logs.

Possible Cause

This issue is likely due to missing customer data in the ECE cache.

Solution

To address this problem:

  1. To understand the customer data that is missing, review the OMC_home/logs/nodeID/dc.log file, where nodeID is the ECE DC node ID.

    For example, you might see something similar to this in the dc.log file:
    2025-10-18 21:56:44;ERROR;Invalid Data; Data not as expected; response from ECE contains reason codes, ECE response with reason code is written to RESPONSE_ERROR_REC file in scratch directory...#EOR#
2025-10-18 21:56:44;INFO ;Number of requests not sent to ECE due to invalid data or some error 0#EOR#
2025-10-18 21:56:45;INFO ;Number of requests not sent to ECE due to invalid data or some error 0#EOR#

  2. Perform a complete restart of ECE to ensure data is reloaded correctly.

  3. Reload the customer data into the ECE cache using incremental customer loading. See "Loading Customer Data Incrementally with customerLoader" in Oracle Communications Billing and Revenue Management System Administrator’s Guide.

  4. Restart Offline Mediation Controller to synchronize the system.

Problem: De-duplication Node Hanging Error

You may find that the Offline Mediation Controller de-duplication Node is hanging, causing Call Detail Records (CDRs) to get stuck.

For example, you might see an error message like this in the logs:
POD xx log of 1 Aggregation Processor:
2024-04-18 22:21:42;ERROR;Initialization; Remote data manager execution; Connection timed out while trying to contact RDM at node-mgr-app-100:55109#EOR#
2024-04-18 22:22:12;ERROR;Initialization; Remote data manager execution; Connection timed out while trying to connect to RDM at node-mgr-app-100:55109#EOR#

Possible Cause

This issue is probably due to memory management problems.

Solution

To address this, try the following:
  1. Configure the Node Manager’s garbage collection threads based on your CPU capacity. For example, set: 
    -XX:ParallelGCThreads=value1
    -XX:ConcGCThreads=value2
    Where:

    • value1 controls the number of threads for stop-the-world phases

    • value2 controls the number of threads for concurrent GC phases

    For on-premises systems, these garbage collection options are to be given in the bin or the nodemgr script. For cloud native environments, set the gcOptions key in the override-values.yaml file.

  2. Reduce the number of keys used to detect duplicate records to only the session ID and the transaction ID for the current operation. You can specify the data used for the duplicate check key in the duplicate check EP node’s rule file. See "About Removing Duplicate Records" in Offline Mediation Controller User’s Guide.

  3. Adjust the JVM heap memory size by increasing it to prevent out-of-memory errors. For more information, see "Configuring Offline Mediation Controller Services" in the Offline Mediation Controller Cloud Native Installation and Administration Guide. For more information about on-premises deployments, see “Configuring Node Manager Memory Limits”.