4 Monitoring PDC REST Services Manager

You can using logs, tracing, metrics, and system health data to monitor Oracle Communications Pricing Design Center (PDC) REST Services Manager.

Topics in this document:

• About Logging
• About Tracing
• About Metrics
• About Monitoring System Health

About Logging

You can review the PDC REST Services Manager logs to troubleshoot errors and monitor system activity.

PDC REST Services Manager uses the Apache Log4j Java logging utility to log information and errors about the following:

• Start up and shut down activity
• Interaction with other applications at integration points while processing publication events. This includes interactions with PDC, Oracle Identity Cloud Service, and your enterprise product catalog.
• Authorization requests
• Authentication requests
• Tracing (see "About Tracing")

You access the logs as the user who installed PDC REST Services Manager. See "Accessing the Logs".

The logs support the standard Log4j logging levels. By default, the framework log levels are set to INFO, and the application log levels are set to DEBUG. You can change the levels after installation. For example, setting the log levels to ALL allows you to log detailed authentication or authorization errors for Helidon security providers. See "Changing the Log Levels".

You can configure the format of the logs. See "Configuring the Log Format."

By default, PDC REST Services Manager routes Java logging to the Log4j log manager. After setting up PDC REST Services Manager, you can change the log manager. See "Changing the Default Log Manager".

For information about Java logging, see "Java Logging Overview" in Java Platform, Standard Edition Core Libraries. For information about Log4j, see https://logging.apache.org/log4j/2.x/manual/index.html.

Oracle recommends using automated log file rotation for PDC REST Services Manager logs. For information about configuring log file rotation, see My Oracle Support article 2087525.1 at https://support.oracle.com/knowledge/Oracle%20Linux%20and%20Virtualization/2087525_1.html.

Accessing the Logs

You access the logs to monitor and troubleshoot your system.

To access the logs:

1. Log in to the system where PDC REST Services Manager was installed as the installation user.
2. Run the following command:

   tail -f PDC_RSM_home/log/rsm.log

   where PDC_RSM_home is the directory in which you installed PDC REST Services Manager.

The following is an example of the logs for updating the 500FreeMinutes product offering:

2020-09-05T20:19:28.399-0700 START RestServicesManager.sh
pdcrsm-6f88869785-vtbw2 pdcrsm 2020-11-13T15:58:06.702Z |  INFO | 9fcdb109-8682-4368-b4d5-b5b720a1af77 | 548aee87-5ef0-4c1a-b8c8-d2b8a8c6fb40 | 500FreeMinutes | 4ca071fde65d2a61 | pool-3-thread-1      | ctPublishEventServiceImpl | Processing Publish Event 548aee87-5ef0-4c1a-b8c8-d2b8a8c6fb40->500FreeMinutes
pdcrsm-6f88869785-vtbw2 pdcrsm 2020-11-13T15:58:07.303Z |  INFO | 9fcdb109-8682-4368-b4d5-b5b720a1af77 | 548aee87-5ef0-4c1a-b8c8-d2b8a8c6fb40 | 500FreeMinutes | 4ca071fde65d2a61 | pool-3-thread-1      | ductOfferingServiceLaunch | Retrieving ProductOffering for ID OOO_DayTech201_OOO
pdcrsm-6f88869785-vtbw2 pdcrsm 2020-11-13T15:58:09.088Z |  INFO | 9fcdb109-8682-4368-b4d5-b5b720a1af77 | 548aee87-5ef0-4c1a-b8c8-d2b8a8c6fb40 | 500FreeMinutes | 4ca071fde65d2a61 | pool-3-thread-1      | .c.b.i.d.PdcRmiConnection | Attempting to connect to PDC using t3s://pdc-service:8002 ...
pdcrsm-6f88869785-vtbw2 pdcrsm Handshake failed: TLSv1.3, error = No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
pdcrsm-6f88869785-vtbw2 pdcrsm Handshake succeeded: TLSv1.2
pdcrsm-6f88869785-vtbw2 pdcrsm 2020-11-13T15:58:12.437Z |  INFO | 9fcdb109-8682-4368-b4d5-b5b720a1af77 | 548aee87-5ef0-4c1a-b8c8-d2b8a8c6fb40 | 500FreeMinutes | 4ca071fde65d2a61 | pool-3-thread-1      | c.b.i.d.PdcDatasourceImpl | Checking if PDC object with the name "500FreeMinutes" exists
pdcrsm-6f88869785-vtbw2 pdcrsm 2020-11-13T15:58:12.479Z |  INFO | 9fcdb109-8682-4368-b4d5-b5b720a1af77 | 548aee87-5ef0-4c1a-b8c8-d2b8a8c6fb40 | 500FreeMinutes | 4ca071fde65d2a61 | pool-3-thread-1      | o.c.b.i.s.PdcServiceImpl  | Updating the PDC object "500FreeMinutes"
pdcrsm-6f88869785-vtbw2 pdcrsm 2020-11-13T15:58:16.134Z |  INFO | 9fcdb109-8682-4368-b4d5-b5b720a1af77 | 548aee87-5ef0-4c1a-b8c8-d2b8a8c6fb40 | 500FreeMinutes | 4ca071fde65d2a61 | pool-3-thread-1      | o.c.b.i.s.PdcServiceImpl  | PDC object successfully updated for "500FreeMinutes"

See "Configuring the Log Format" for information about the format of the logs.

Changing the Log Levels

You can change the global log level and the level for PDC REST Services Manager application-specific log entries.

To change the log levels:

1. Open the PDC_RSM_home/apps/conf/logging-config.yaml file in a text editor, where PDC_RSM_home is the directory in which you installed PDC REST Services Manager.
2. To change the global log level:
   1. Search for the following property: -name: ROOT_LOG_LEVEL.
   2. Set the value to "${env:ROOT_LOG_LEVEL:-LEVEL}", where LEVEL is the new log level.
3. To change the PDC REST Services Manager application log level:
   1. Search for the following property: -name: PDC_RSM_LOG_LEVEL.
   2. Set the value to "${env:PDC_RSM_LOG_LEVEL:-LEVEL}", where LEVEL is the new log level.
4. Save and close the file.
5. Restart PDC REST Services Manager using the control script located in the bin directory:

   PDC_RSM_home/apps/bin/RestServicesManager.sh restart

The following shows sample entries in the logging-config.yaml file, with the relevant lines and default values in bold:

Configuration:
  name: Default
 
  Properties:
    Property:
      - name: ROOT_LOG_LEVEL
        value: "${env:ROOT_LOG_LEVEL:-INFO}"
      - name: PDC_RSM_LOG_LEVEL
        value: "${env:PDC_RSM_LOG_LEVEL:-INFO}"
 
  Appenders:
    Console:
      name: LogToConsole
      target: SYSTEM_OUT
      PatternLayout:
        Pattern: "%d{ISO8601_OFFSET_DATE_TIME_HHCMM} | %5p | %X{eventId} | %X{projectId} | %X{productOfferId} | %X{traceId} | %-20.20thread | %-25.25logger{25} | %m%n"
 
  loggers:
    Root:
      level: "${ROOT_LOG_LEVEL}"
      AppenderRef:
        - ref: LogToConsole
 
 
    Logger:
      - name: io.jaegertracing.internal.JaegerSpan
        level: error
        AppenderRef:
          - ref: LogToConsole
      - name: io.jaegertracing.internal.reporters
        level: warn
        AppenderRef:
          - ref: LogToConsole
      - name: oracle.communications
        level: "${PDC_RSM_LOG_LEVEL}"
        additivity: false
        AppenderRef:
          - ref: LogToConsole

Note:

The Zipkin tracing logs that appear under Logger are filtered from the logs, so you do not need to adjust their levels.

Configuring the Log Format

Configure the log format to change the order and number of elements that appear in the logs.

The default log format is:

%d{ISO8601_OFFSET_DATE_TIME_HHCMM} | %5p | %X{eventId} | %X{projectId} | %X{productOfferId} | %X{traceId} | %-20.20thread | %-25.25logger{25} | %m%n

where:

• %d is the date and time of the log, in ISO 8601 format.
• %5p is the log level. See "Changing the Log Levels".
• eventId, projectId, productOfferId, and traceId are tags added for tracing events and objects through the system. See "Using Trace Tags to Troubleshoot Issues".
• %-20.20thread is the thread pool for the logged event.
• %-25.25logger is the service logging the event.
• %m%n is the message associated with the event.

To configure the log format:

1. Open the PDC_RSM_home/apps/conf/logging-config.yaml file in a text editor, where PDC_RSM_home is the directory in which you installed PDC REST Services Manager.
2. Under Appenders, locate the PatternLayout property.
3. In the value for Pattern, change the order of the elements or remove any unwanted elements.

   Note:

   Removing elements from the logs might reduce your ability to troubleshoot issues and trace messages.
4. Save and close the file.
5. Restart PDC REST Services Manager using the control script located in the bin directory:

   PDC_RSM_home/apps/bin/RestServicesManager.sh restart

Changing the Default Log Manager

By default, PDC REST Services Manager uses the Log4J Log Manager. You can change this after configuring PDC REST Services Manager.

To change the log manager, run the following command:

Java_home/bin/java -Djava.util.logging.manager=logManager

where:

• Java_home is the directory where you installed the latest compatible version of Java.
• logManager is the log manager you want to use. By default, this is set to org.apache.logging.log4j.jul.LogManager when you install PDC REST Services Manager. To use your system default, leave this empty.

About Tracing

You can trace the flow of messages through PDC REST Services Manager by using the Zipkin tracer integrated with the Helidon framework, or another transaction tracing tool of your choice.

Helidon is a collection of Java libraries used by PDC REST Services Manager. Zipkin is an open-source tracing system integrated with Helidon. You can use the Zipkin interface to monitor the PDC REST Services Manager traces.

Figure 4-1 shows an example of tracing an event through the system in the Zipkin tracer.

Figure 4-1 Example Event Tracing

Description of "Figure 4-1 Example Event Tracing"

For more information about Helidon and Zipkin, see:

• The Helidon project website: https://helidon.io/

• The discussion of Zipkin tracing in the Helidon documentation: https://helidon.io/docs/v2/#/se/tracing/02_zipkin

• The Zipkin website: https://zipkin.io/

To set up tracing in PDC REST Services Manager:

1. Do one of the following:

   • Install Zipkin. See the Zipkin Quickstart documentation: https://zipkin.io/pages/quickstart.html.

   • Integrate Zipkin tracing with Helidon SE. See the Helidon SE Tracing Guide documentation: https://helidon.io/docs/v2/#/se/guides/06_tracing.

2. Enable Zipkin tracing in PDC REST Services Manager. See "Enabling Tracing in PDC REST Services Manager".

3. Optionally, add trace tags to help troubleshoot and trace messages and objects through the system. See "Using Trace Tags to Troubleshoot Issues".

Afterward, you can start tracing the flow of messages by using the Zipkin UI or Zipkin API.

Enabling Tracing in PDC REST Services Manager

To enable tracing:

1. Configure tracing in your Helidon project, as described in the Helidon documentation: https://helidon.io/docs/latest/#/mp/tracing/01_tracing.
2. Open the PDC_RSM_home/apps/conf/application.yaml file in a text editor, where PDC_RSM_home is the directory in which you installed PDC REST Services Manager.
3. Under tracing, set enabled to true.
   The following is a sample of the tracing section in the application.yaml file:

   tracing:  
       enabled: true
       service: "rsm"  
       host: "localhost" 
       paths:
           - path: "/metrics"      
             enabled: false
       propagation: "zipkin"  
       log-spans: true  
       max-queue-size: 42  
       flush-interval-ms: 10001

4. Save and close the application.yaml file.
5. Restart PDC REST Services Manager by running the following command from the PDC_RSM_home/apps/bin directory:
   ./RestServicesManager.sh restart

Using Trace Tags to Troubleshoot Issues

Instead of reading through logs to identify and troubleshoot issues, you can use trace tags in PDC REST Services Manager to correlate logs and traces.

PDC REST Services Manager tags events with the following trace tags:

• publishId: A general tag for the event. In the example below, this is the first id.
• eventId: A tag for the event that is specific to PDC REST Services Manager. In the example below, this is the eventId.
• projectId: A tag for the project in the enterprise product catalog. In the example below, this is the ID under project.
• productOfferId: A tag for a product offering. In the example below, this is the ID under each entry in the projectItems array.
• productSpecificationId: A tag for product specifications. This does not appear in the example below, but would appear in log messages. You use the productOfferId tag to filter logs and locate related productSpecificationId tags as needed.

The following shows an example event for publishing updates to two product offerings from an enterprise product catalog to PDC. To illustrate an error scenario, a URL in the payload for the testInit4Offer product offering has become corrupt. The IDs corresponding to trace tags are shown in bold.

{
  "id": "d64066bd-2954-4f43-b8f2-69603c88c683",
  "eventId": "ea09ae5a-8098-4fb2-b634-ee8048b9cc1d",
  "eventTime": "2030-11-18T09:31:50.001Z",
  "eventType": "projectPublishEvent",
  "correlationId": "UC4Fcfc6a70f-60f5-456c-93d5-d8e038215201",
  "domain": "productCatalogManagement",
  "timeOcurred": "2030-11-18T09:31:50.001Z",
  "event": {
    "project": {
      "id": "demopackage11",
      "lifecycleStatus": "IN_DESIGN",
      "name": "Project01",
      "acknowledgementUrl": "http://host:port/mobile/custom/PublishingAPI",
      "projectItems": [
        {
          "id": "55c8362b32d36b49",
          "href": "http://host:port/mobile/custom/catalogManagement/productOffering/testSuccess",
          "name": "testSuccess",
          "version": "1.0",
          "@referredType": "ProductOfferingOracle"
        },
        {
          "id": "55c8362b32d36b55",
          "href": "http://host:port/mobile/custom/CORRUPTDATA/productOffering/testInit4Offer",
          "name": "100Minutes",
          "version": "1.0",
          "@referredType": "ProductOfferingOracle"
        }
      ]
    }
  }
}

Trace Tags in Tracer Tools

After submitting the event, you can follow its progress and look for the trace tags in a tracer tool like Zipkin.

Figure 4-2 shows excerpts from a tracer. You can immediately see that the error occurred in the GET request of the getProductOfferingDetails operation. You can expand the trace spans to get the IDs for the event and the object in question, then search in the logs for those tags, as well as the span and trace IDs, to troubleshoot the issue.

Figure 4-2 Sample Tracer Excerpts

Description of "Figure 4-2 Sample Tracer Excerpts"

The following is the same data for listenToProjectPublishEvent and getProductOfferingDetails in JSON format, with the relevant IDs in the tags arrays in bold:

{
  "traceID": "55c8362b32d36b49",
  "spanID": "bad2ef5f3ff26084",
  "flags": 1,
  "operationName": "listenToProjectPublishEvent",
  "references": [
    {
      "refType": "CHILD_OF",
      "traceID": "f2f902949ee8e661",
      "spanID": "8ce5e8f8cda38d3b"
    }
  ],
  "startTime": 1605709909244000,
  "duration": 18160,
  "tags": [
    {
      "key": "eventId",
      "type": "string",
      "value": "ea09ae5a-8098-4fb2-b634-ee8048b9cc1d"
    },
    {
      "key": "http.status_code",
      "type": "int64",
      "value": 201
    },
    {
      "key": "component",
      "type": "string",
      "value": "jaxrs"
    },
    {
      "key": "span.kind",
      "type": "string",
      "value": "server"
    },
    {
      "key": "http.url",
      "type": "string",
      "value": "http://host:port/productCatalogManagement/v1/projectPublishEvent"
    },
    {
      "key": "http.method",
      "type": "string",
      "value": "POST"
    },
    {
      "key": "projectId",
      "type": "string",
      "value": "demopackage11"
    },
    {
      "key": "publishId",
      "type": "string",
      "value": "d64066bd-2954-4f43-b8f2-69603c88c683"
    },
    {
      "key": "internal.span.format",
      "type": "string",
      "value": "Zipkin"
    }
  ],
  "logs": [],
  "processID": "p1",
  "warnings": null
},
...
{
  "traceID": "f2f902949ee8e661",
  "spans": [
    {
      "traceID": "f2f902949ee8e661",
      "spanID": "03031b1c18e679f2",
      "flags": 1,
      "operationName": "getProductOfferingDetails",
      "references": [
        {
          "refType": "CHILD_OF",
          "traceID": "f2f902949ee8e661",
          "spanID": "528a32ac350706e2"
        }
      ],
      "startTime": 1605709909256000,
      "duration": 688729,
      "tags": [
        {
          "key": "productOfferId",
          "type": "string",
          "value": "testInit4Offer"
        },
        {
          "key": "internal.span.format",
          "type": "string",
          "value": "Zipkin"
        }
      ],
      "logs": [],
      "processID": "p1",
      "warnings": null
    },
    {
      "traceID": "f2f902949ee8e661",
      "spanID": "303707dcd9c9d1ef",
      "flags": 1,
      "operationName": "getProductOfferingDetails",
      "references": [
        {
          "refType": "CHILD_OF",
          "traceID": "f2f902949ee8e661",
          "spanID": "d1d2c068248a5542"
        }
      ],
      "startTime": 1605709909277000,
      "duration": 529234,
      "tags": [
        {
          "key": "error",
          "type": "bool",
          "value": true
        },
        {
          "key": "productOfferId",
          "type": "string",
          "value": "testInit4Offer"
        },
        {
          "key": "internal.span.format",
          "type": "string",
          "value": "Zipkin"
        }
      ],
      "logs": [
        {
          "timestamp": 1605709909807000,
          "fields": [
            {
              "key": "event",
              "type": "string",
              "value": "error"
            },
            {
              "key": "error.object",
              "type": "string",
              "value": "oracle.communications.brm.integration.exceptions.EccServiceException"
            }
          ]
        }
      ],
      "processID": "p1",
      "warnings": null
    }
  ]
}

Trace Tags in Logs

After finding the trace tags in the tracer tool, you can search the logs for them. You can do simple searches in the raw log data, or you can search and filter by the tags using a logging tool, such as Grafana Loki.

The trace tags appear in the following format in PDC REST Service Manager logs:

yyyy-MM-dd'T'HH:mm:ss.SSSXXX, UTC | level | eventId | projectId | productOfferId | traceId | thread | logging service | message

The following shows the success message in the logs for updating the testInit4Offer product, with the relevant trace tags from the event in bold:

2030-10-11T11:34:36,231+05:30 | INFO | ea09ae5a-8098-4fb2-b634-ee8048b9cc1d | demopackage11 | testInit4Offer | 55c8362b32d36b49 | pool-4-thread-1 | ctPublishEventServiceImpl | Processing Publish Event ea09ae5a-8098-4fb2-b634-ee8048b9cc1d->testInit4Offer

For the testInit4Offer product, the following error log appears:

2020-11-18T14:31:49.814Z | ERROR | ea09ae5a-8098-4fb2-b634-ee8048b9cc1d | demopackage11 | testInit4Offer | f2f902949ee8e661 | pool-3-thread-4 | .s.LaunchPdcItemPublisher | Error calling API service 'Product Offering Service' for 'testInit4Offer'. Status Code: 404 Error: '

Based on this message and what you saw in the tracer, you would know that PDC REST Services Manager wasn't able to make the call to the enterprise product catalog to request information about the testInit4Offer product offering. Expanding and inspecting the GET span in the tracer would reveal the corrupt URL. You could then review the message that came from your enterprise product catalog to confirm, and make appropriate changes to resolve the issue.

About Metrics

You can monitor the PDC REST Services Manager metrics by using the Metrics REST endpoint. The metrics count successful and failed messages passing through the PDC REST Services Manager integration points.

Use a monitoring tool that scrapes metrics data, such as Prometheus, to monitor the metrics available from the PDC REST Services Manager Metrics endpoint. You can get the metrics in plain text format, which is compatible with Prometheus, or JSON format. See "Checking Access to Metrics" for information about accessing the metrics endpoint and requesting different formats. For more information about Prometheus, see: https://prometheus.io/.

Table 4-1 shows the available metrics.

Table 4-1 PDC REST Services Manager Metrics

Integration Point	Metric	Description
PDC interface	pdc-create-object-success-total	The number of create events that returned a success from PDC.
PDC interface	pdc-create-object-error-total	The number of create events that returned an error from PDC.
PDC interface	pdc-update-object-success-total	The number of update events that returned a success from PDC.
PDC interface	pdc-update-object-error-total	The number of update events that returned an error from PDC.
Product Offer Price Project life cycle event listener	notification-listener-change-success-total	The number of well-formed publish events received by PDC REST Services Manager.
Product Offer Price Project life cycle event listener	notification-listener-change-error-total	The number of publish events accepted by PDC REST Services Manager that could not be processed due to invalid or incomplete event payloads.
Product Offering interface	product-offering-get-success-total	The number of Product Offering GET API requests that returned a success from the enterprise product catalog.
Product Offering interface	product-offering-get-error-total	The number of Product Offering GET API requests that returned an error from the enterprise product catalog.
Product Specification interface	product-specification-get-success-total	The number of Product Specification GET API requests that returned a success from the enterprise product catalog.
Product Specification interface	product-specification-get-error-total	The number of Product Specification GET API requests that returned an error from the enterprise product catalog.
Publish Notification interface	publish-job-status-success-total	The number of Publish Notification Acknowledgments that returned a success from the enterprise product catalog.
Publish Notification interface	publish-job-status-fail-total	The number of Publish Notification POST Acknowledgments that returned an error from the enterprise product catalog.
Publish Product Offering service	publish-product-offering-success-total	The number of successful Product Offering Publish actions.
Publish Product Offering service	publish-product-offering-fail-total	The number of Failed Product Offering Publish actions.

You can also use built-in Helidon metrics. See the Helidon documentation for more information: https://helidon.io/docs/latest/#/mp/metrics/01_introduction.

Checking Access to Metrics

You can access the PDC REST Services Manager metrics from any tool that can access REST API endpoints using an OAuth token generated by Oracle Identity Cloud Service for PDC REST Services Manager. You can check whether you have access by using cURL commands.

To check whether you have access to the PDC REST Services Manager metrics:

1. In the command line on the system where cURL and your scraping tool are installed, export your OAuth access token with the following command:

   export TOKEN=OAuth_metrics_token

   where OAuth_metrics_token is the token you requested for the Metrics endpoint in "Requesting an OAuth Access Token from Oracle Identity Cloud Service".

2. Enter one of the following commands:
   • To get the metrics in plain text format:

     curl --insecure -H "Authorization: Bearer $TOKEN" https://hostname:port/metrics

     where:
     • hostname is the URL for the PDC REST Services Manager server.
     • port is the TLS port for the PDC REST Services Manager server.
   • To get the metrics in JSON format:

     curl --insecure -H "Authorization: Bearer $TOKEN" -H "Accept: application/json" https://hostname:port/metrics

About Monitoring System Health

You can assess the health of the PDC REST Services Manager system by monitoring the process status and overall rates of failure in logs, traces, and metrics.

To monitor process status, use the control script in PDC_RSM_home/apps/bin directory, where PDC_RSM_home is the directory in which you installed PDC REST Services Manager:

RestServicesManager.sh status

To maintain an active system, Oracle recommends using a service from your operating system, such as systemd on Linux, to automatically start, monitor, and restart the PDC REST Services Manager system.