11 Using Logs to Monitor Components

Learn how to use logs to monitor your Oracle Communications Billing and Revenue Management (BRM) cloud native deployment.

Topics in this document:

Overview of BRM Cloud Native Logging

BRM cloud native services write log messages to standard output and standard error. Kubernetes captures these container output streams and makes them available as pod logs. However, Kubernetes will rotate and remove log files when they get to a certain size/number, so for longer-term storage, you will want a log management system. See https://kubernetes.io/docs/concepts/cluster-administration/logging/#log-rotation in the Kubernetes documentation for information on setting the size and number of these files.

The logging process is as follows:

  1. The BRM service writes log messages to standard output and standard error.

  2. The container runtime captures the container's stdout/stderr, and the kubelet stores the resulting logs on the worker node under /var/log/containers/ by default.

  3. While the logs are stored in the container runtime, you can view them using Kubernetes commands such as kubectl logs. See "Viewing Component Logs" for information.

  4. A log forwarder, such as Fluentd, collects the container logs from the worker node, parses multiline BRM log entries, enriches the records with Kubernetes metadata, and forwards them to the log management system. See "Configuring Log Collection and Forwarding" for information.

  5. A log management system stores, searches, visualizes, and generates alerts for the logs. See "Configuring the Log Management System" for information.

Note:

BRM includes Fluentd as the pre-integrated log forwarder, but also supports other log forwarders. BRM supports external log management systems, but does not provide pre-integration for a specific system.

Viewing Component Logs

You can use Kubernetes commands in Table 11-1 to view logs for BRM pods.

Table 11-1 Kubernetes Commands for Viewing BRM Component Logs

Command Description
kubectl get pods -n BrmNamespace

Lists the pods in the BRM namespace.
kubectl logs PodName -n BrmNamespace

Displays logs for the specified BRM pod.
kubectl logs -f PodName -n BrmNamespace

Displays logs for the specified BRM pod and continues to stream new log messages.
kubectl logs PodName -c ContainerName -n BrmNamespace

Displays logs for a specific container in the specified BRM pod.

where:

  • PodName is the name of the BRM pod.

  • ContainerName is the name of the container in the pod.

  • BrmNamespace is the namespace in which you deployed BRM.

Configuring Log Collection and Forwarding

To manage the logs and store them for longer time periods, configure a log forwarding system like Fluentd and a log management system.

The log forwarding system performs the following functions:

  1. Collects logs from container output streams.

  2. Parses logs by using configured patterns for BRM components.

  3. Enriches log records with Kubernetes metadata, such as namespace, pod name, and container name.

  4. Forwards logs to a log management system.

Using the Sample Fluentd Configuration Files

The BRM cloud native Helm chart provides the following sample Fluentd configuration files:

oc-cn-helm-chart/Fluentd-klog-patterns-configmap.yaml: Configures Fluentd to process Kubernetes container logs in K-log format.

oc-cn-helm-chart/Fluentd-json-patterns-configmap.yaml: Configures Fluentd to process logs in JSON format.

Each sample file includes a source configuration that reads Kubernetes container logs for the BRM namespace. In the <source> section, update the path and tag keys to refer to the BRM namespace. For more information, see https://docs.fluentd.org/configuration/config-file#list-of-directives.

Use the <match> sections in the file to configure the destination for the log files for your selected log management system, for example:

<match **>
  @type forward
  <server>
    host log-collector.example.com
    port 24224
  </server>
</match>

Note:

Review and update these sample configuration files for your environment before you use them.

Configuring the Log Management System

Caution:

The container runtime rotates and deletes log files on worker nodes. Configure a log management system if you need long-term log retention, search capabilities, alerting, audit records, or troubleshooting history.

Configure a log management system to manage BRM logs that your log forwarding system sends.

The log management system performs the following functions:

  • Receives logs from the configured log forwarding system.

  • Stores and indexes log records.

  • Provides search and visualization capabilities.

  • Generates alerts based on configured rules, such as warning or error patterns.

  • Applies retention, access control, and compliance policies.

You can use any log management system that meets your operational, storage, security, and compliance requirements.

Configure the log management system to receive logs from the log forwarding system.

Setting the Reporting Level for Logging Messages

By default, BRM components report error messages, and BRM applications report both error and warning messages (level 2). The four levels of error reporting are:

  • 0 = No logging

  • 1 = Log error messages only

  • 2 = Log error messages and warnings

  • 3 = Log error, warning, and debugging messages

Note:

To avoid performance degradation, use log level 3 only for debugging.

A common implementation is to set the reporting level to 2 and use the configured log management system to detect warning messages and generate alerts.

To change the severity level for logging:

  1. Open your override-values.yaml file for oc-cn-helm-chart.

  2. Edit the log level entry for the BRM component or application. For information about product-specific logging configuration settings, see "Configuring BRM Server and PDC Services" in BRM Cloud Native Deployment Guide.

  3. Save and close the file.

  4. Stop and restart the component or application.