Using Logs to Monitor Components

BRM records system activity in log files. One log file is generated for each component or application. Review these files daily to monitor your system and detect and diagnose system problems. You can also:

Write scripts to look for certain conditions, such as types or numbers of errors, and to notify you when these conditions occur.
Record opcode calls in the CM log file. See "Recording Opcode Calls in the CM Log File".

BRM generates log files for system components, applications, and client applications.

For system processes (or threads) such as CMs and DMs, BRM uses two types of log files:

Those that record normal startup activity are named program.log (for example, cm.log, js.log, and dm.log).
Those that record activity, such as error conditions, while the system is running. These pinlogs are named program.pinlog (for example, cm.pinlog, js.pinlog, and dm_oracle.pinlog).

For BRM applications, log files are named program.pinlog (for example, pin_billd.pinlog). If an application is missing a configuration file (pin.conf) or if the application fails before it can read the configuration file, it records errors in the default.pinlog log file.

Note:

Calls made by opcodes to get data from objects are not recorded in log files.

BRM Java-based applications, such as Customer Center and Configuration Center, by default do not use log files. However, you can enable error logging by adding entries to the Infranet.properties file that provide configuration information when the application starts.

Log files for system components are stored in BRM_home/sys/component. For example, the CM log file is in BRM_home/sys/cm.

If there is no log file in BRM_home/var/component, the default.pinlog file is used instead. It is stored in BRM_home/sys/component. For example, the CM pinlog file is BRM_home/sys/cm/default.pinlog.

For an application or client application log file, the default location is the directory from which the program was started.

You can leave log files in their default locations or move them.

Large log files degrade system performance. Check the sizes of log files periodically and delete or archive large files. When you delete or rename a log file, a new empty file is created as soon as a new log entry is created and either a maximum of four hours have elapsed or the application is stopped and restarted. Be especially vigilant when using new custom applications, which commonly makes log files grow quickly.

Log files should be archived weekly to a safe storage area.

Changing the Name or Location of a Log File

To change the name or location of the pinlog file for a component or application:

Open the configuration file (pin.conf or Infranet.properties) for the component or application.
Change the relevant entry:
- logfile: Applications
- cm_logfile: CM
- dm_logfile: DM
Enter the desired name and directory for the log file.
Save and close the file.
Stop and restart the component or application.

Note:

You can change the name of the default application's log file by using the PIN_ERR_SET_LOGFILE function. See "PIN_ERR_SET_LOGFILE" in BRM Developer's Reference.

Setting the Reporting Level for Logging Messages

By default, BRM components report error messages, and BRM applications report both error and warning messages. You can set BRM to report debugging messages or not to report errors. The four levels of error reporting are:

0 = No logging
1 = (Default) Log error messages only
2 = Log error messages and warnings
3 = Log error, warning, and debugging messages

Note:

To avoid performance degradation, use only level 3 logging for debugging.

A common implementation is to set the error-level reporting to 2 and use a script to detect and act on warning messages.

To change the severity level for logging:

Open the component's or application's configuration file (pin.conf or .properties).
Edit the loglevel entry. The notes in the configuration file define the options.
Save and close the file.
Stop and restart the component or application.

Setting the Log Level for a Specific Opcode

You can record debug-level information for a specified opcode without having to reset the default system log level. This enables you to monitor the activity of a specific opcode (and any opcode it calls) without impacting system performance.

When you enable opcode logging, the logging level is increased to debug level 3 for the specified opcode only; all other opcodes are logged at the level specified in the CM pin.conf file.

You can define how many times during a CM session the debug-level reporting occurs for the specified opcode before the default reporting level is restored. This enables you to increase the logging level without having to stop and restart the CM to reset it to the default level.

Open the CM pin.conf file in BRM_home/sys/cm.
Set the pinlog_debug_opcode entry:
```
cm pinlog_debug_opcode opcode
```
where opcode is the opcode name or opcode number.

Note:

If this entry is not set, BRM uses the loglevel entry in the CM pin.conf file to determine the log level.
Set the pinlog_debug_op_count entry:
```
cm pinlog_debug_op_count number
```
where number is the number of times the opcode is recorded at the debug level before the default log level is restored.
Save and close the file.
Restart the CM.

Recording Opcode Calls in the CM Log File

You use the enable_pcm_op_call_stack and max_pcm_op_call_stack_entries CM pin.conf entries to record opcodes in the CM log file.

When enable_pcm_op_call_stack is enabled, the opcodes that are called by BRM clients are recorded in the CM log file.

See "Connection Manager (CM) pin.conf Entries".

About Formatting Log Files

You can format a log file to improve readability and traceability of errors by using the splitPinlog script. This script splits a log file into multiple files, one for each combination of process ID (PID) and thread ID (TID) based on the information in the header of the pinlog entries.

To format a log file:

Go to the BRM_home/bin directory.
Run the following Perl script:
```
splitPinlog original_pinlog_file
```

The Perl script creates a file with the name original_pinlog_file.pid.tid.pinlog.

For example, running the command:

splitPinlog cm.pinlog

results in these file names:

cm.pinlog.342353.12.pinlog
cm.pinlog.342353.13.pinlog

Using an EFK Stack for Log Files

You can use an Elasticsearch, Fluentd, and Kibana (EFK) stack as a centralized logging solution for BRM. In this configuration, Fluentd formats the BRM log files and then forwards them to Elasticsearch for storage. Kibana provides a GUI dashboard that you use to view, search, and analyze the log data.

To use an EFK stack, see the following:

Creating an EFK Stack for BRM

Creating an EFK stack for BRM involves these high-level steps:

Install and run Elasticsearch. You can start Elasticsearch manually or run it as a daemon or service.
See "Installing Elasticsearch" and "Starting Elasticsearch" in the Elasticsearch documentation.
Install Kibana. See "Install Kibana" in the Kibana documentation.
Open the config/kibana.yml file and set the following values:
- server.host: Set this to the name of the server host where you want to run Kibana.
- elasticsearch.hosts: Set this to the name of the server hosts where you run Elasticsearch.
Run Kibana. You can start Kibana manually or run it as a service.
See "Start and stop Kibana" in the Kibana documentation.
Install Fluentd by using the td-agent RPM package, launch the td-agent daemon, and post sample logs to verify the installation. See "Install by RPM Package" in the Fluentd documentation.
Install the Fluentd concatenation plugin by entering the following command:
```
/usr/sbin/td-agent-gem install fluent-plugin-concat
```

Configure the /etc/td-agent/td-agent.conf file to collect logs from BRM and send them to Elasticsearch. The following sample td-agent.conf file is configured to collect and send the BRM CM logs, with the path to the CM logs specified in the source block. You can also add source and filter blocks for the Data Manager (dm_oracle) and EAI Java Server (eai_js) logs.

<match fluent.**>
  @type null
</match>
 
<source>
  @type tail
  @id in_tail_CM_logs
  path BRM_home/var/cm/cm.pinlog
  pos_file /var/log/fluentd-test.log.pos
  tag TEST.*
  read_from_head true
  <parse>
    @type none
  </parse>
</source>
 
<filter TEST**>
  @type concat
  @id cm_concat
  stream_identity_key pod_name
  key message
  multiline_start_regexp /^(?<severity>\w+)\s+\D{3}\s+\D{3}\s+\d{1,2}/
  flush_interval 5
  timeout_label @NORMAL
</filter>
 
<match **>
  @type relabel
  @label @NORMAL
</match>
 
<label @NORMAL>
  <match **>
    @type elasticsearch
    @id out_es
    @log_level info
    include_tag_key true
    host localhost
    port 9200
    scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
    ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
    ssl_version "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERSION'] || 'TLSv1'}"
    reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'false'}"
    reconnect_on_error "#{ENV['FLUENT_ELASTICSEARCH_RECONNECT_ON_ERROR'] || 'true'}"
    reload_on_failure "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_ON_FAILURE'] || 'true'}"
    log_es_400_reason "#{ENV['FLUENT_ELASTICSEARCH_LOG_ES_400_REASON'] || 'false'}"
    logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
    logstash_format "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT'] || 'true'}"
    index_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME'] || 'test-logstash'}"
    type_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_TYPE_NAME'] || 'test-fluentd'}"
    <buffer>
      flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '8'}"
      flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '5s'}"
      chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '2M'}"
      queue_limit_length "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH'] || '32'}"
      retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}"
      retry_forever true
    </buffer>
  </match>
</label>

Note:

The host in the Elasticsearch match block is localhost, which assumes Elasticsearch and Fluentd are running on the same host. In a production environment, they would be running on different hosts, and you would have to specify the Elasticsearch host instead.

See "Post Installation Guide" and "Config File Syntax" in the Fluentd documentation for more information about using the configuration file.

Restart td-agent after editing td-agent.conf.

Configuring Elasticsearch in a Production Environment

By default, Elasticsearch binds to the local host for HTTP and internal transport communication. In a clustered production environment, you need to change this setting.

When configuring a production system, do the following:

When configuring the node for Elasticsearch, set the following properties in the config/elasticsearch.yml file:
- network.bind_host: Set this to 0.0.0.0. This lets Elasticsearch use the addresses of all available network interfaces rather than a loopback address.
- http.port: Set this to 9200.
- discovery.seed_hosts: Set this to [ ] (You can leave an empty array, or optionally provide a comma-separated list of seed hosts if applicable to your deployment.)
See "Networking" and "Important Elasticsearch configuration" in the Elasticsearch documentation for more information about these settings.

If you see bootstrap check failures when starting Elasticsearch, confirm that you have configured the settings for production described in "Important System Configuration" in the Elasticsearch documentation.
Configure Kibana in the same node as Elasticsearch.
Configure Fluentd in a different node. Use the IP address or host name of the Elasticsearch node in the match block.

See "Bootstrapping a cluster" in the Elasticsearch documentation for more information about setting up Elasticsearch in a production cluster.

Accessing Logs in the Kibana Dashboard

To access BRM logs in the Kibana dashboard:

Use a web browser to go to the following URL:
```
 http://kibana_host:5601/app/kibana
```
where kibana_host is the host where you run Kibana, specified in the server.host property in the kibana.yml file.
Confirm that you have at least one index. From Kibana’s main menu, select Stack Management and then select Index Management.
If there are no indices, create a data view as described in "Create a Data view" in the Kibana documentation.
From Kibana's main menu, select Discover. Your BRM log data appears on the dashboard.

Example of Using Kibana to Filter BRM Logs

This example shows how to use Kibana to filter BRM error logs and identify correlated errors.

To use Kibana to see a set of correlated BRM errors:

From Kibana's main menu, select Discover.
Add the following filters:
- _index: logstash_name where logstash_name is the name of the logstash you want to see.
  For example, logstash-2021-01-01.
- message: is one of E
  This filter shows errors of severity E only. See "Understanding Error-Message Syntax" for more information about BRM error logs and severity levels.
Find the correlation ID of the error of interest. For example:
```
2:CT1255:Account_Manager:1948:1684:63:1063403309:14:root.0.0.0.1:::. 
```
See "Logging External User Information in Error Logs" for more information about the correlation ID format.
Add the following filter:
- message: is one of correlation_id
where correlation_id is the correlation ID you located above.

All BRM errors with the selected correlation ID are shown.