1. Cloud Native System Administrator's Guide
  2. Administering PDC Cloud Native Services
  3. Monitoring PDC REST Services Manager

20 Monitoring PDC REST Services Manager

Learn how to monitor Oracle Communication Pricing Design Center (PDC) REST Services Manager in a cloud native environment using logging, tracing, metrics, and system health data.

Topics in this document:

About PDC REST Services Manager Logs

You can review the PDC REST Services Manager logs to troubleshoot errors and monitor system activity.

PDC REST Services Manager uses the Apache Log4j Java logging utility to log information and errors about the following:
  • Start up and shut down activity
  • Interaction with other applications at integration points while processing publication events. This includes interactions with PDC, Oracle Identity Cloud Service, and your master product catalog.
  • Authorization requests
  • Authentication requests
  • Zipkin tracing (see "About PDC REST Services Manager Tracing")

You access the logs in the Cloud Native BRM environment using the kubectl command in the BRM namespace. See "Accessing the PDC REST Services Manager Logs".

The logs support the standard Java logging levels. By default, the log levels are set to INFO. You can change the levels after installation. For example, setting the log levels to ALL allows you to log detailed authentication or authorization errors for Helidon security providers. See "Changing the Log Levels".

By default, PDC REST Services Manager routes Java logging to the Log4j log manager. After setting up PDC REST Services Manager, you can change the log manager. See "Changing the Default Log Manager Using Helm".

For general information about Java logging, see Java Platform, Standard Edition Core Libraries. For information about Log4j, see: https://logging.apache.org/log4j/2.x/manual/index.html

Oracle recommends using automated log file rotation for PDC REST Services Manager logs. For information about configuring log file rotation, see My Oracle Support article 2087525.1 at: https://support.oracle.com/knowledge/Oracle%20Linux%20and%20Virtualization/2087525_1.html

Accessing the PDC REST Services Manager Logs

You access the PDC REST Services Manager logs to monitor and troubleshoot your system.

To access the logs:

  1. To get the names of the PDC REST Services Manager pods, enter this command:

    kubectl -n BRMNameSpace get pods | grep pdcrsm
    The following is an example of the command's output, with the pod names in bold:
    pdcrsm-7f48565595-bndp8                  1/1     Running   0          6h35m
pdcrsm-7f48565595-hqfwb                  1/1     Running   0          6h35m

  2. To access the logs, enter this command:

    kubectl -n BRMNameSpace logs PDCRSMPodName

    where PDCRSMPodname is the name of the PDC REST Services Manager pod you want the log for.

    The following is an example of the logs for updating the 500FreeMinutes product offering: 

    pdcrsm-6f88869785-vtbw2 pdcrsm 2020-11-13T15:58:06.702Z | INFO | 9fcdb109-8682-4368-b4d5-b5b720a1af77 | 548aee87-5ef0-4c1a-b8c8-d2b8a8c6fb40 | 500FreeMinutes | 4ca071fde65d2a61 | pool-3-thread-1      | ctPublishEventServiceImpl | Processing Publish Event 548aee87-5ef0-4c1a-b8c8-d2b8a8c6fb40->500FreeMinutes
pdcrsm-6f88869785-vtbw2 pdcrsm 2020-11-13T15:58:07.303Z | INFO | 9fcdb109-8682-4368-b4d5-b5b720a1af77 | 548aee87-5ef0-4c1a-b8c8-d2b8a8c6fb40 | 500FreeMinutes | 4ca071fde65d2a61 | pool-3-thread-1      | ductOfferingServiceLaunch | Retrieving ProductOffering for ID 500FreeMinutes
pdcrsm-6f88869785-vtbw2 pdcrsm 2020-11-13T15:58:09.088Z | INFO | 9fcdb109-8682-4368-b4d5-b5b720a1af77 | 548aee87-5ef0-4c1a-b8c8-d2b8a8c6fb40 | 500FreeMinutes | 4ca071fde65d2a61 | pool-3-thread-1      | .c.b.i.d.PdcRmiConnection | Attempting to connect to PDC using t3s://pdc-service:8002 ...
pdcrsm-6f88869785-vtbw2 pdcrsm Handshake failed: TLSv1.3, error = No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
pdcrsm-6f88869785-vtbw2 pdcrsm Handshake succeeded: TLSv1.2
pdcrsm-6f88869785-vtbw2 pdcrsm 2020-11-13T15:58:12.437Z | INFO | 9fcdb109-8682-4368-b4d5-b5b720a1af77 | 548aee87-5ef0-4c1a-b8c8-d2b8a8c6fb40 | 500FreeMinutes | 4ca071fde65d2a61 | pool-3-thread-1      | c.b.i.d.PdcDatasourceImpl | Checking if PDC object with the name "500FreeMinutes" exists
pdcrsm-6f88869785-vtbw2 pdcrsm 2020-11-13T15:58:12.479Z | INFO | 9fcdb109-8682-4368-b4d5-b5b720a1af77 | 548aee87-5ef0-4c1a-b8c8-d2b8a8c6fb40 | 500FreeMinutes | 4ca071fde65d2a61 | pool-3-thread-1      | o.c.b.i.s.PdcServiceImpl  | Updating the PDC object "500FreeMinutes"
pdcrsm-6f88869785-vtbw2 pdcrsm 2020-11-13T15:58:16.134Z | INFO | 9fcdb109-8682-4368-b4d5-b5b720a1af77 | 548aee87-5ef0-4c1a-b8c8-d2b8a8c6fb40 | 500FreeMinutes | 4ca071fde65d2a61 | pool-3-thread-1      | o.c.b.i.s.PdcServiceImpl  | PDC object successfully updated for "500FreeMinutes"

Note:

This task shows how to access a single log at a time. To tail logs from multiple pods, Oracle recommends using the Kubernetes Stern tool. See the Stern repository for more information: https://github.com/stern/stern

Changing the Log Levels

You can change the root log level and the level for PDC REST Services Manager application-specific log entries either by changing Helm values or by editing the PDC REST Services Manager Kubernetes deployment resource.

For a more permanent solution, use Helm, which requires upgrading the Helm deployment. See "Changing the Log Levels Using Helm".

For quicker troubleshooting, use Kubernetes. See "Changing the Log Levels Using Kubernetes".

Changing the Log Levels Using Helm

Change the log levels using Helm for longer-term logging.

To change the log levels using Helm:
  1. In the override-values.yaml file, under the entry for ocpdcrsm, edit the values for rootLoglevel and appLogLevel as needed.
    The following is an example of the ocpdcrsm entry, with the default values of INFO in bold:
    ocpdcrsm:
    isEnabled: true
    labels:
        name: "pdcrsm"
        version: "15.0.x.0.0"
    deployment:
        deadlineSeconds: 60
        revisionHistLimit: 10
        imageName:
          pdcrsm: "oracle/pdcrsm"
        # For non-empty tag, ":" MUST be prepended
        imageTag: ":15.0.x.0.0"
        imagePullPolicy: IfNotPresent
        rootLoglevel: INFO
        appLogLevel: INFO
  2. Update your Helm release. See "Updating a Helm Release".
Changing the Log Levels Using Kubernetes

Change the log levels using Kubernetes for short-term troubleshooting logging.

To change the log levels using Kubernetes:
  1. Enter this command:
    kubectl -n BRMNameSpace set env deployment/pdcrsm ROOT_LOG_LEVEL=level PDC_RSM_LOG_LEVEL=level

    where level is the log level you want to set.

    The following is an example of the ocpdcrsm entry, with the default values of INFO in bold:
    ocpdcrsm:
    isEnabled: true
    labels:
        name: "pdcrsm"
        version: "15.0.x.0.0"
    deployment:
        deadlineSeconds: 60
        revisionHistLimit: 10
        imageName:
          pdcrsm: "oracle/pdcrsm"
        # For non-empty tag, ":" MUST be prepended
        imageTag: ":15.0.x.0.0"
        imagePullPolicy: IfNotPresent
        rootLoglevel: INFO
        appLogLevel: INFO
  2. Update your Helm release. See "Updating a Helm Release".

Note:

Next time a Helm update is performed, changes made using Kubernetes will be overwritten. If you want to make the change permanent, update the Helm override-values.yaml file as described in "Changing the Log Levels Using Helm".

Changing the Default Log Manager Using Helm

By default, PDC REST Services Manager uses the Log4J Log Manager. You can change this after configuring PDC REST Services Manager.

To change the log manager using Helm:

  1. In the override-values.yaml file, under the entry for ocpdcrsm, edit the value for -Djava.util.logging.manager= in JAVA_OPTS.

    By default, this is set to org.apache.logging.log4j.jul.LogManager when you install PDC REST Services Manager. To use your system default, leave -Djava.util.logging.manager= empty, as in the following example. 

    ocpdcrsm:
    isEnabled: true
    labels:
        name: "pdcrsm"
        version: "15.0.x.0.0"
    deployment:
        deadlineSeconds: 60
        revisionHistLimit: 10
        imageName:
          pdcrsm: "oracle/pdcrsm"
        # For non-empty tag, ":" MUST be prepended
        imageTag: ":15.0.x.0.0"
        imagePullPolicy: IfNotPresent
        rootLoglevel: ALL
        appLogLevel: ALL
        JAVA_OPTS: -Djava.util.logging.manager=
  2. Update your Helm release. See "Updating a Helm Release".

About PDC REST Services Manager Tracing

You can trace the flow of REST API calls made to PDC REST Services Manager using Zipkin, an open-source tracing system. For more information, see the Zipkin website: https://zipkin.io/.

To set up tracing in PDC REST Services Manager cloud native:

  1. Install Zipkin. See the Zipkin Quickstart documentation: https://zipkin.io/pages/quickstart.html.

  2. Enable Zipkin tracing in PDC REST Services Manager cloud native. See "Enabling Tracing in PDC REST Services Manager".

  3. Optionally, add trace tags to help troubleshoot and trace messages and objects through the system. See "Using Trace Tags to Troubleshoot Issues".

Afterward, you can start tracing the flow of REST API calls made to PDC REST Services Manager using the Zipkin UI or Zipkin API.

Enabling Tracing in PDC REST Services Manager

By default, tracing is disabled in PDC REST Services Manager cloud native, but you can enable it at any time.

To enable tracing with Zipkin:

  1. In the override-values.yaml file for oc-cn-helm-chart, set ocpdcrsm.configEnv.isTracingEnabled to true.
  2. Run the helm upgrade command to update your Helm release:
    helm upgrade BrmReleaseName oc-cn-helm-chart --values OverrideValuesFile -n BrmNameSpace

    where:

    • BrmReleaseName is the release name for oc-cn-helm-chart and is used to track this installation instance.

    • OverrideValuesFile is the file name and path to your override-values.yaml file.

    • BrmNameSpace is the namespace in which to create BRM Kubernetes objects for the BRM Helm chart.

Using Trace Tags to Troubleshoot Issues

Instead of reading through logs to identify and troubleshoot issues, you can use trace tags in PDC REST Services Manager to correlate logs and traces.

PDC REST Services Manager tags events with the following trace tags:

  • publishId: A general tag for the event. In the example below, this is the first id.
  • eventId: A tag for the event that is specific to PDC REST Services Manager. In the example below, this is the eventId.
  • projectId: A tag for the project in the enterprise product catalog. In the example below, this is the ID under project.
  • productOfferId: A tag for a product offering. The example below shows the ID under each entry in the projectItems array.
  • productSpecificationId: A tag for product specifications. This does not appear in the example below but would appear in log messages. You use the productOfferId tag to filter logs and locate related productSpecificationId tags as needed.

The following shows an example event for publishing updates to two product offerings from an enterprise product catalog to PDC. To illustrate an error scenario, a URL in the payload for the testInit4Offer product offering has become corrupt. The IDs corresponding to trace tags are shown in bold. 

{
  "id": "d64066bd-2954-4f43-b8f2-69603c88c683",
  "eventId": "ea09ae5a-8098-4fb2-b634-ee8048b9cc1d",
  "eventTime": "2030-11-18T09:31:50.001Z",
  "eventType": "projectPublishEvent",
  "correlationId": "UC4Fcfc6a70f-60f5-456c-93d5-d8e038215201",
  "domain": "productCatalogManagement",
  "timeOcurred": "2030-11-18T09:31:50.001Z",
  "event": {
    "project": {
      "id": "demopackage11",
      "lifecycleStatus": "IN_DESIGN",
      "name": "Project01",
      "acknowledgementUrl": "http://host:port/mobile/custom/PublishingAPI",
      "projectItems": [
        {
          "id": "55c8362b32d36b49",
          "href": "http://host:port/mobile/custom/catalogManagement/productOffering/testSuccess",
          "name": "testSuccess",
          "version": "1.0",
          "@referredType": "ProductOfferingOracle"
        },
        {
          "id": "55c8362b32d36b55",
          "href": "http://host:port/mobile/custom/CORRUPTDATA/productOffering/testInit4Offer",
          "name": "100Minutes",
          "version": "1.0",
          "@referredType": "ProductOfferingOracle"
        }
      ]
    }
  }
}

Trace Tags in Tracer Tools

After submitting the event, you can follow its progress and look for the trace tags in a tracer tool like Zipkin.

Figure 20-1 shows excerpts from a tracer. You can immediately see that the error occurred in the GET request of the getProductOfferingDetails operation. You can expand the trace spans to get the IDs for the event and the object in question, then search in the logs for those tags, as well as the span and trace IDs, to troubleshoot the issue.

Figure 20-1 Sample Tracer Excerpts


Description of Figure 20-1 follows
Description of "Figure 20-1 Sample Tracer Excerpts"

The following is the same data for listenToProjectPublishEvent and getProductOfferingDetails in JSON format, with the relevant IDs in the tags arrays in bold:

{
  "traceID": "55c8362b32d36b49",
  "spanID": "bad2ef5f3ff26084",
  "flags": 1,
  "operationName": "listenToProjectPublishEvent",
  "references": [
    {
      "refType": "CHILD_OF",
      "traceID": "f2f902949ee8e661",
      "spanID": "8ce5e8f8cda38d3b"
    }
  ],
  "startTime": 1605709909244000,
  "duration": 18160,
  "tags": [
    {
      "key": "eventId",
      "type": "string",
      "value": "ea09ae5a-8098-4fb2-b634-ee8048b9cc1d"
    },
    {
      "key": "http.status_code",
      "type": "int64",
      "value": 201
    },
    {
      "key": "component",
      "type": "string",
      "value": "jaxrs"
    },
    {
      "key": "span.kind",
      "type": "string",
      "value": "server"
    },
    {
      "key": "http.url",
      "type": "string",
      "value": "http://host:port/productCatalogManagement/v1/projectPublishEvent"
    },
    {
      "key": "http.method",
      "type": "string",
      "value": "POST"
    },
    {
      "key": "projectId",
      "type": "string",
      "value": "demopackage11"
    },
    {
      "key": "publishId",
      "type": "string",
      "value": "d64066bd-2954-4f43-b8f2-69603c88c683"
    },
    {
      "key": "internal.span.format",
      "type": "string",
      "value": "Zipkin"
    }
  ],
  "logs": [],
  "processID": "p1",
  "warnings": null
},
...
{
  "traceID": "f2f902949ee8e661",
  "spans": [
    {
      "traceID": "f2f902949ee8e661",
      "spanID": "03031b1c18e679f2",
      "flags": 1,
      "operationName": "getProductOfferingDetails",
      "references": [
        {
          "refType": "CHILD_OF",
          "traceID": "f2f902949ee8e661",
          "spanID": "528a32ac350706e2"
        }
      ],
      "startTime": 1605709909256000,
      "duration": 688729,
      "tags": [
        {
          "key": "productOfferId",
          "type": "string",
          "value": "testInit4Offer"
        },
        {
          "key": "internal.span.format",
          "type": "string",
          "value": "Zipkin"
        }
      ],
      "logs": [],
      "processID": "p1",
      "warnings": null
    },
    {
      "traceID": "f2f902949ee8e661",
      "spanID": "303707dcd9c9d1ef",
      "flags": 1,
      "operationName": "getProductOfferingDetails",
      "references": [
        {
          "refType": "CHILD_OF",
          "traceID": "f2f902949ee8e661",
          "spanID": "d1d2c068248a5542"
        }
      ],
      "startTime": 1605709909277000,
      "duration": 529234,
      "tags": [
        {
          "key": "error",
          "type": "bool",
          "value": true
        },
        {
          "key": "productOfferId",
          "type": "string",
          "value": "testInit4Offer"
        },
        {
          "key": "internal.span.format",
          "type": "string",
          "value": "Zipkin"
        }
      ],
      "logs": [
        {
          "timestamp": 1605709909807000,
          "fields": [
            {
              "key": "event",
              "type": "string",
              "value": "error"
            },
            {
              "key": "error.object",
              "type": "string",
              "value": "oracle.communications.brm.integration.exceptions.EccServiceException"
            }
          ]
        }
      ],
      "processID": "p1",
      "warnings": null
    }
  ]
}

Trace Tags in Logs

After finding the trace tags in the tracer tool, you can search the logs for them. You can do simple searches in the raw log data or search and filter by the tags using a logging tool, such as Grafana Loki.

The trace tags appear in the following format in PDC REST Service Manager logs:

yyyy-MM-dd'T'HH:mm:ss.SSSXXX, UTC | level | eventId | projectId | productOfferId | traceId | thread | logging service | message

The following shows the success message in the logs for updating the testInit4Offer product, with the relevant trace tags from the event in bold: 

2030-10-11T11:34:36,231+05:30 | INFO | ea09ae5a-8098-4fb2-b634-ee8048b9cc1d | demopackage11 | testInit4Offer | 55c8362b32d36b49 | pool-4-thread-1 | ctPublishEventServiceImpl | Processing Publish Event ea09ae5a-8098-4fb2-b634-ee8048b9cc1d->testInit4Offer

For the testInit4Offer product, the following error log appears: 

2020-11-18T14:31:49.814Z | ERROR | ea09ae5a-8098-4fb2-b634-ee8048b9cc1d | demopackage11 | testInit4Offer | f2f902949ee8e661 | pool-3-thread-4 | .s.LaunchPdcItemPublisher | Error calling API service 'Product Offering Service' for 'testInit4Offer'. Status Code: 404 Error: '

Based on this message and what you saw in the tracer, you would know that PDC REST Services Manager couldn't call the enterprise product catalog to request information about the testInit4Offer product offering. Expanding and inspecting the GET span in the tracer would reveal the corrupt URL. You could then review the message from your enterprise product catalog to confirm and make appropriate changes to resolve the issue.

About PDC REST Services Manager Metrics

You can monitor the PDC REST Services Manager metrics by using the Metrics REST endpoint. The metrics count successful and failed messages passing through the PDC REST Services Manager integration points.

Use a monitoring tool that scrapes metrics data, such as Prometheus, to monitor the metrics available from the PDC REST Services Manager Metrics endpoint. You can get the metrics in plain text format, which is compatible with Prometheus, or JSON format. See "Checking Access to PDC REST Services Manager Metrics" for information about accessing the metrics endpoint and requesting different formats. For more information about Prometheus, see: https://prometheus.io/.

Table 20-1 shows the available PDC REST Services Manager metrics.

Table 20-1 PDC REST Services Manager Metrics

Integration Point Metric Description
PDC interface pdc-create-object-success-total The number of Create events that returned a success from PDC.
PDC interface pdc-create-object-error-total The number of Create events that returned an error from PDC.
PDC interface pdc-update-object-success-total The number of update events that returned a success from PDC.
PDC interface pdc-update-object-error-total The number of update events that returned an error from PDC.
Product Offer Price Project life cycle event listener notification-listener-change-success-total The number of well-formed publish events received by PDC REST Services Manager.
Product Offer Price Project life cycle event listener notification-listener-change-error-total The number of publish events accepted by PDC REST Services Manager that could not be processed due to invalid or incomplete event payloads.
Product Offering interface product-offering-get-success-total The number of Product Offering GET API requests that returned a success from the master product catalog.
Product Offering interface product-offering-get-error-total The number of Product Offering GET API requests that returned an error from the master product catalog.
Product Specification interface product-specification-get-success-total The number of Product Specification GET API requests that returned a success from the master product catalog.
Product Specification interface product-specification-get-error-total The number of Product Specification GET API requests that returned an error from the master product catalog.
Publish Notification interface publish-job-status-success-total The number of Publish Notification Acknowledgments that returned a success from the master product catalog.
Publish Notification interface publish-job-status-fail-total The number of Publish Notification POST Acknowledgments that returned an error from the master product catalog.
Publish Product Offering service publish-product-offering-success-total The number of successful Product Offering Publish actions.
Publish Product Offering service publish-product-offering-fail-total The number of Failed Product Offering Publish actions.

You can also use Helidon framework metrics. See the Helidon documentation for more information: https://helidon.io/docs/v1/#/metrics/01_metrics.

Checking Access to PDC REST Services Manager Metrics

You can access the PDC REST Services Manager metrics from any tool that can access REST API endpoints using an OAuth token generated by Oracle Identity Cloud Service for PDC REST Services Manager. You can check whether you have access by using cURL commands.

To check whether you have access to the PDC REST Services Manager metrics:
  1. In the command line on the system where cURL and your scraping tool are installed, export your OAuth access token with the following command: 
    export TOKEN=OAuth_metrics_token

    where OAuth_metrics_token is the client secret you stored for the Metrics scope in "Configuring OAuth Authentication in PDC REST Services Manager" in BRM Cloud Native Deployment Guide.

  2. Enter one of the following commands:
    • To get the metrics in plain text format:
      curl --insecure -H "Authorization: Bearer $TOKEN" https://hostname:port/metrics
      where:
      • hostname is the URL for the PDC REST Services Manager server.
      • port is the TLS port for the PDC REST Services Manager server.
    • To get the metrics in JSON format: 
      curl --insecure -H "Authorization: Bearer $TOKEN" -H "Accept: application/json" https://hostname:port/metrics

About Monitoring PDC REST Services Manager System Health

You can assess the health of the PDC REST Services Manager system by monitoring the pod status and using the Health REST endpoint.

See:

Verifying the PDC REST Services Manager Pod Status

To verify the pod status, run this command:

kubectl -n BRMNameSpace get pods --selector=app.kubernetes.io/name=pdcrsm
The following is an example of the command output:
NAME                             READY     STATUS      RESTARTS       AGE
pdcrsm-b9d7bb7d6-j2xsl7          1/1       Running     0              105m
pdcrsm-b9d7bb7d6-lfxcl           1/1       Running     0              105m

Note:

Kubernetes provides automatic health monitoring and will attempt to restart applications when they fail.

Using the PDC REST Services Manager Health Endpoint

You can monitor overall system health by submitting a GET request to the following endpoint:

https://hostname:port/health

where:

  • hostname is the URL for the PDC REST Services Manager server
  • port is the TLS port for the PDC REST Services Manager server

The response contains information about:

  • Deadlocked threads
  • Disk space used
  • Memory heap used

The following is an example of the response:

{
    "outcome": "UP",
    "status": "UP",
    "checks": [
        {
            "name": "deadlock",
            "state": "UP",
            "status": "UP"
        },
        {
            "name": "diskSpace",
            "state": "UP",
            "status": "UP",
            "data": {
                "free": "101.80 GB",
                "freeBytes": 109306679296,
                "percentFree": "69.01%",
                "total": "147.52 GB",
                "totalBytes": 158399414272
            }
        },
        {
            "name": "heapMemory",
            "state": "UP",
            "status": "UP",
            "data": {
                "free": "399.05 MB",
                "freeBytes": 418431544,
                "max": "6.89 GB",
                "maxBytes": 7393378304,
                "percentFree": "99.41%",
                "total": "440.88 MB",
                "totalBytes": 462290944
            }
        }
    ]
}