1. Administrator's Guide
  2. Administrative Procedures
  3. Monitoring the Store

Monitoring the Store

You can obtain Information about the performance and availability of your store from both server side and client side perspectives:

  • Your Oracle NoSQL Database applications can obtain performance statistics using the oracle.kv.KVStore.getStats() class. This provides a client side view of the complete round trip performance for Oracle NoSQL Database operations.

  • Oracle NoSQL Database automatically captures Replication Node performance statistics into a log file that you can into into spreadsheet software for analysis. The store tracks, logs, and writes statistics at a user specified interval to a CSV file. The file is je.stat.csv, located in the Environment directory. Logging occurs per-Environment when the Environment is opened in read/write mode.

    Configuration parameters control the size and number of rotating log files to use (similar to java logging, see java.util.logging.FileHandler). For a rotating set of files, as each file reaches a given size limit, it is closed, rotated out, and a new file is opened. Successively older files are named with an incrementing numeric suffix to the file name. The name format is je.stat[version].csv.

  • The Oracle NoSQL Database administrative service collects and aggregates status information, alerts, and performance statistics components that the store generates. This provides a detailed view of the behavior and performance of the Oracle NoSQL Database server.

  • Each Oracle NoSQL Database Storage Node maintains detailed logs of trace information from the services that the node supports. The administrative service presents an aggregated, store-wide view of these component logs. Logs are available on each Storage Node if the administrative service is not available, or if it is more convenient to examine individual Storage Node logs. Additionally, you can compress these log files to store more logging output in the same disk space.

  • Oracle NoSQL Database supports the optional Java Management Extensions (JMX) agents for monitoring. The JMX interfaces allow you to poll the Storage Nodes for information about the storage node and any replication nodes that it hosts. For more information on JMX monitoring, see Standardized Monitoring Interfaces. For information on using JMX securely, see Guidelines for using JMX securely in the Security Guide.

You can monitor the status of the store by verifying it from within the CLI. See Verifying the Store. You can also use the CLI to examine events.

Events

Events are special messages that inform you of the state of your system. As events are generated, they are routed through the monitoring system so that you can see them. There are four types of events that the store reports:

  1. State Change events are issued when a service starts up or shuts down.

  2. Performance events report statistics about the performance of various services.

  3. Log events are records produced by the various system components to provide trace information about debugging. These records are produced by the standard java.util.logging package.

  4. Plan Change events record the progress of plans as they execute, are interrupted, fail or are canceled.

Note:

  • Some events are considered critical. These events are recorded in the administration service's database, and can be retrieved and viewed using the CLI.
  • You can compress the log event records that are produced by the standard java.util.logging package. For more information, see Log File Compression

You cannot view Plan Change events directly through Oracle NoSQL Database's administrative interfaces. However, State Change events, Performance events, and Log events are recorded using the EventRecorder facility, which is internal to the Admin. Only events considered critical are recorded, and the criteria for being designated as such varies with the event type. These are the events considered critical:

  • All state changes.
  • Log events classified as SEVERE.
  • Any performance events reported as below a certain threshold.

You can view all of these critical events using the Admin CLI show events and show event commands.

Use the CLI show events command with no arguments to see all of the unexpired events in the database. Use the -from and -to arguments to limit the range of events that display. Use the -type or -id arguments to filter events by type or id, respectively.

For example, this is part of the output from a show events command: 

kv-> show events
idarpdfbS STAT 2015-08-13 22:18:39.287 UTC sn1 RUNNING sev1
idarpeg0S STAT 2015-08-13 22:18:40.608 UTC sn2 RUNNING sev1
idarphmuS STAT 2015-08-13 22:18:44.742 UTC rg1-rn1 RUNNING sev1
idarpjLLS STAT 2015-08-13 22:18:47.289 UTC rg1-rn2 RUNNING sev1
idartfcuS STAT 2015-08-13 22:21:48.414 UTC rg1-rn2 UNREACHABLE sev2
                                                   (reported by admin1)

This result shows four service state change events (sev1) and one log event (UNREACHABLE), classified as sev2. Tags at the beginning of each line are individual event record identifiers. To see detailed information for a particular event, use the show event command, which takes an event record identifier, such as idartfcuS as its argument: 

kv-> show event -id idartfcuS
idartfcuS STAT 2015-08-13 22:21:48.414 UTC rg1-rn2 UNREACHABLE sev2
  (reported by admin1)

Using this method of event identifiers, you can see a complete stack trace.

Events are removed from the system if the total number of events is greater than a set maximum number, or if the Event is older than a set period. The default maximum number of events is 10,000, while the default time period is 30 days.

Both Sev1 and Sev2 flags are associated with specific service state change events. Sev1 flags report the current state. Sev2 flags report errors during attempted state changes, as follows:

Sev1 Flags Sev2 Flags
STARTING ERROR_RESTARTING
WAITING_FOR_DEPLOY ERROR_NO_RESTART
RUNNING UNREACHABLE
STOPPING  
STOPPED  

Log File Compression

You can compress the log files by setting the serviceLogFileCompression Storage Node parameter. This helps to store significantly more logging output in the same amount of disk space. As a result, the logs can be retained for a longer period, rendering their availability for debugging purposes. The log files are compressed with the standard gzip algorithm. By default, the compression of log files stores approximately five times more data as compared with uncompressed files.

When you enable the log file compression, restart the associated Storage Node Agent for the new setting to take effect on the new files and compress the existing files. The size of log files may temporarily exceed the defined limits in certain cases. The rotated copies of the store-combined debug logs, statistics files, and performance files are compressed. Additionally, the service debug log files across all Storage Nodes, Replication Nodes, Admins, and Arbiters are compressed.

The compressed rotated log files are renamed with a .gz suffix and the file name is modified to use the creation time instead of the standard log rotation number.

For example:

Precompressed log file Post compressed log file
rg1-rn1_1.log rg1-rn1_20220720-201902.log.gz

Here, the standard rotated log file rg1-rn1_1.log is renamed to rg1-rn1_20220720-201902.log.gz after compression. The date part of the log file implies that the file was created on 2022-07-20 20:19:02 UTC.

If multiple rotated log files are created in the same second, a unique suffix is added to the date portion of the file name.

For example: rg1-rn1_20220720-201902-1.log.gz.

Note:

  • To avoid the usage of extra disk space in the standard log directory, you can use the gzcat command to view the contents without uncompressing the zipped files. Use the zgrep command to search the compressed log files. You can also uncompress the files into another directory. Manually uncompressed files are not deleted automatically.
  • The Bootstrap debug log files, GC log files, JE debug log files, and JE statistics files are not compressed.