34 Troubleshooting BRM Cloud Native

Learn how to troubleshoot issues in Oracle Communications Billing and Revenue Management (BRM) cloud native.

Topics in this document:

Authorization Error When Running pin_update_balances Utility

Problem Statement

When running the pin_update_balances utility, you may encounter an authorization error and the process fails to complete.

Sample Error Message

E Wed May 15 12:16:35 2026 <hostname> pin_update_balances:2579170 pin_mta.c:1076 3:<hostname>:pin_update_balances:2579170:1778034432:4:1715764595851079:8:bill_inv_pymt_sub.0.0.0.1::::-1 normal error <location=PIN_ERRLOC_FM:5 class=PIN_ERRCLASS_SYSTEM_DETERMINATE:1 errno=PIN_ERR_AUTHORIZATION_FAIL:64> <facility=0 msg_id=0 version=0>

Cause

This issue occurs because the user bill_inv_pymt_sub.0.0.0.1 is not authorized to run the PCM_OP_CUST_UPDATE_ROLE opcode.

Solution

  1. Update the following entries in the testnap ConfigMap file (configmap_pin_conf_testnap.yaml).

    Note:

    Do not change these entries in the CM ConfigMap file (configmap_pin_conf_cm.yaml).

    Original entries:

    - nap login_name root.0.0.0.1
- - userid 0.0.0.1 /service/pcm_client 1

    Update it to:

    - nap login_name super_user.0.0.0.1
- - userid 0.0.0.1 /service/pcm_client 3

  2. Run the helm upgrade command to update your Helm release.

  3. Update the flist for PCM_OP_CUST_UPDATE_ROLE:

    0 PIN_FLD_POID POID [0] 0.0.0.1 /config/role 412 0
0 PIN_FLD_PERMITTEDS ARRAY [0] allocated 20, used 3
1   PIN_FLD_BASE_OPCODES STR [0] ""
1   PIN_FLD_CLASSES STR [0] ""
1   PIN_FLD_FM_OPCODES STR [0] "PCM_OP_ACT_REQUEST_CREATE,PCM_OP_ACT_REQUEST_RETRIEVE,PCM_OP_ACT_USAGE,PCM_OP_BILL_MAKE_BILL"
0 PIN_FLD_PERMITTEDS ARRAY [3] allocated 20, used 3
1   PIN_FLD_BASE_OPCODES STR [0] "PCM_OP_READ_FLDS"
1   PIN_FLD_CLASSES STR [0] "/billinfo,/payinfo/*,/item/misc,/account,/config/payment,/bill,/event/*,/data/trial_bill_sequence,/item/installment/debit,/ledger_"
1   PIN_FLD_FM_OPCODES STR [0] ""

  4. Run the PCM_OP_CUST_UPDATE_ROLE opcode using testnap. For more information on how to run testnap in cloud native environments, see "Running Load Utilities Through Configurator Jobs".

  5. Return the entries in the configmap_pin_conf_testnap.yaml file to their original values: 

    - nap login_name root.0.0.0.1
- - userid 0.0.0.1 /service/pcm_client 1

  6. Run the helm upgrade command to update your Helm release.

Order Processing Fails with Kafka DM Connection Errors

Problem Statement

Order processing fails intermittently when the Connection Manager (CM) cannot connect to the Kafka DM to publish events. The CM log contains messages similar to the following:

bad cm_change_dm_connection, new db=<DB>, current db=<DB>,
err=26(PIN_ERR_DM_CONNECT_FAILED), errno=9(Bad file descriptor)

Op (PCM_OP_TRANS_OPEN) send failed in pcm_op

The connection failure does not resolve automatically.

Cause

This problem can occur when the EAI Java Server (JS) and the Kafka DM use different thread pool sizes. Over time, this mismatch can lead to thread exhaustion and connection failures.

Solution

To fix this, do the following:

  1. Verify the connection pool settings in the EAI Java Server (JS) source configuration in the infranet.properties file of the eai_js module used for your deployment.

  2. Set the infranet.connection.pool.enable entry to true. Record the value of the infranet.connection.pool.size entry. If you change either entry, apply the updated EAI Java Server configuration before continuing.

  3. Open the override-values.yaml file for oc-cn-helm-chart.

  4. Set the ocbrm.dm_kafka.deployment.poolSize key to the value of the infranet.connection.pool.size entry that you recorded for the EAI Java Server.

    Use a matching value from 1 through 2000 that meets your deployment requirements.

  5. Save and close the file.

  6. Run the following command to update the BRM Helm release:

    helm upgrade BrmReleaseName oc-cn-helm-chart \
    --values OverrideValuesFile --namespace BrmNameSpace

    where:

    • BrmReleaseName is the release name for oc-cn-helm-chart.

    • OverrideValuesFile is the file name and path of the override-values.yaml file.

    • BrmNameSpace is the namespace that contains the BRM Kubernetes objects.

  7. Retrieve the BRM pod names:

    kubectl get pods -n BrmNameSpace

  8. In the command output, identify the Kafka DM pod name.

  9. Delete the Kafka DM pod to restart it:

    kubectl delete pod DmKafkaPodName -n BrmNameSpace

    Kubernetes creates a replacement Kafka DM pod.

  10. Verify that the replacement Kafka DM pod has a status of Running:

    kubectl get pods -n BrmNameSpace

  11. Retry the failed operation. Verify that the CM log no longer reports PIN_ERR_DM_CONNECT_FAILED.

    Note:

    Restarting the Kafka DM pod without correcting the thread pool mismatch restores service only temporarily.

    For more information, see "Integrating with Kafka Servers" and "Restarting BRM Pods".

Grafana Dashboards are Empty

Problem Statement

The sample Grafana dashboards included in the oc-cn-docker-files-16.26.0.x.x.tgz package do not display data.

Cause

This issue can occur if ServiceMonitor resources are missing, not selected by Prometheus, or configured with labels or namespaces that do not match the Prometheus configuration. It can also occur when the BRM services are exposing metrics, but the collected metrics do not include labels such as pod or namespace that are used by the dashboard filters.

Solution

To fix this issue:

  1. In the Prometheus UI, confirm that the BRM targets are listed and healthy.

  2. If you use Prometheus Operator, confirm that the ServiceMonitor resources exist for the CM and dm-oracle metrics and that they are selected by Prometheus.

  3. Verify that the BRM services expose the /metrics endpoint.

  4. In Grafana, confirm that dashboard query filters such as pod and namespace match the labels available in Prometheus for the collected metrics.

    If the BRM metrics are available in Prometheus but not shown in Grafana, the dashboard queries may be filtering on labels that are missing or different from the labels in the Prometheus metrics.

SSL/TLS Handshake Error When Connecting to CM Through an External Load Balancer

Problem Statement

A BRM client fails to connect to CM through an external load balancer. The connection fails during the SSL/TLS handshake.

Cause

This issue can occur when the client does not trust the certificate presented by the load balancer, or when the load balancer uses a plain TCP connection to connect to an SSL/TLS-enabled CM.

Solution

  1. Verify that the Oracle wallet used by the client trusts the certificate presented by the load balancer.

  2. Verify that the host name or IP address in the certificate matches the load balancer host name or IP address used by the client.

    If the load balancer connects to CM using SSL/TLS, verify that the load balancer is configured to start an SSL/TLS connection to CM. See “Configuring an External Load Balancer for Connection Manager SSL/TLS Connections” for instructions.