Monitoring for Storage Nodes (SN)

A Storage Node is a physical (or virtual) machine with its own local storage, which houses the Replication Node. For more information, see Architecture in the Concepts Guide.

See the following sections:

Metrics for Storage Nodes

snServiceStatus – The current status of the Storage Node Agent running on the host. The Storage Node Agent manages all the Replication Nodes running on the Storage Node (host). The textual representation along with the enumeration ID are shown below:
- starting (1) – The Storage Node Agent is booting up.
- waitingForDeploy (2) – The Storage Node Agent is waiting for the initial deploy-SN command to be run.
- running(3) – The Storage Node Agent is running.
- stopping(4) – The Storage Node Agent is in the process of shutting down. It may be in the process of shutting down Replication Nodes that it manages.
- stopped(5) – An intentional clean shutdown.
- errorRestarting(6) – Although this state exists in the category, it is typically never seen for storage node agents.
- errorNoRestart(7) – Although this state exists in the category, it is typically never seen for storage node agents.
- unreachable(8) – The Storage Node Agent is unreachable by the admin service.
Note: If a Storage Node is UNREACHABLE, or a Replication Node is having problems and its Storage Node is UNREACHABLE, first check the network connectivity between the Admin and the Storage Node. If the managing Storage Node Agent is reachable, but the managed Replication Node is not, the problem most likely lies somewhere other than the network.
- expectedRestarting(9) – This state is rare for Storage Node Agents.
snHostName – The name of the host where the Storage Node agent has been deployed.
snRegistryPort – The TCP/IP port on which Oracle NoSQL Database should be contacted.
snHAHostName – If the HA host name has been configured through the boot parameters then this is returned, otherwise the name of the host running the Storage Node agent is returned. This value represents the network interface name that the replication subsystem uses for internode communication. The HA host name is specified using the -hahost flag to the makebootconfig command, and it corresponds to the haHostname Storage Node parameter, in the Setting Store Parameters. If users encounter a problem indicating that the HA host name has been specified incorrectly, first check that they have used the correct value in the call to the makebootconfig command. The user can change the value later by modifying the haHostname parameter. For more information, see makebootconfig.
snHaPortRange – The range of ports that replication nodes use to communicate among themselves.
snStoreName – The name of the KVStore that this storage node agent is servicing.
snRootDirPath – The fully qualified path to the root of the directory structure where the Oracle NoSQL Database installation files exist.
snLogFileCount – A logging config parameter that represents the maximum number of log files that are retained by the Storage Node Agent.
snLogFileLimit – A logging config parameter that represents the maximum size of a single log file in bytes.
snCapacity – The current capacity of the Storage Node. This parameter essentially describes the number of persistent storage devices on the Storage Node and is typically set at store initialization time, but can be modified by the administrator if the hardware configuration is changed after the store is initialized.
snMountPoints – A list of one or more fully qualified paths to the data files that reside on this storage node.
snMemory – The current memory configuration for this Storage Node in megabytes. This parameter is typically set at store initialization time, but can be modified by the administrator if the hardware configuration is changed after the store is initialized.
snCPUs – The current number of CPUs configured for this Storage Node. This parameter is typically set at store initialization time. The administrator can modify the number if the hardware configuration changes after the store is initialized.
snCollectorInterval – The interval that all nodes are using for aggregate statistics.

Java Management Extensions (JMX) Notifications

Mbean Object Name: Oracle NoSQL Database:type=StorageNode

New operation performance metrics are available as follows:

Type: oracle.kv.repnode.opmetric
User Data: Contains a full listing of performance metrics for a given RN. The statistics are a string in JSON form, and are obtained via Notification.getUserData().

These metrics contain statistics of each type of API operation. And each operation statistics is calculated by interval and cumulative statistics. Interval statistics cover a single measurement period, cumulative statistics cover the duration of this repNode’s uptime. Statistics follows the following naming convention:
```
[Operation]_[Interval|Cumulative]_[Metric]
```

[Operation] has following user operations: Gets, Puts, PutIfAbsent, PutIfPresent, PutIfVersion, Deletes, DeleteIfVersion, MultiGets, MultiGetKeys, MultiGetIterator, MultiGetKeysIterator, StoreIterator, StoreKeysIterator, MultiDeletes, Executes, IndexIterator, IndexKeysIterator, QuerySinglePartition, QueryMultiPartition, QueryMultiShard, BulkPut, BulkGet, BulkGetKeys, BulkGetTable, BulkGetTableKeys

AllSingleKeyOperations are Gets, Puts, PutIfAbsent, PutIfPresent, PutIfVersion, Deletes, DeleteIfVersion

AllMultiKeyOperations are MultiGetKeys, MultiGetIterator, MultiGetKeysIterator, StoreIterator, StoreKeysIterator, MultiDeletes, Executes, IndexIterator, IndexKeysIterator, QuerySinglePartition, QueryMultiPartition, QueryMultiShard, BulkPut, BulkGet, BulkGetKeys, BulkGetTable, BulkGetTableKeys

Read operations are Gets, MultiGets, MultiGetKeys, MultiGetIterator, MultiGetKeysIterator, StoreIterator, StoreKeysIterator, IndexIterator, IndexKeysIterator, QuerySinglePartition, QueryMultiPartition, QueryMultiShard, BulkGet, BulkGetKeys, BulkGetTable, BulkGetTableKeys

Write operation are Puts, PutIfAbsent, PutIfPresent, PutIfVersion, Deletes, DeleteIfVersion, MultiDeletes, Executes, BulkPut

[Metric] has the following types:

TotalReq: The total number of operation requests.
TotalOps: The total number of records returned or processed. Single operation requests only apply to one record, but the multi, iterate, query, bulk or execute operation requests will work on multiple records.
PerSec: Operation throughput per second, that is [TotalOps] / [Interval]
Min: minimum latency
Max: maximum latency
Avg: average latency
95th: The maximum value within the bottom 95% of latency values.
99th: The maximum value within the bottom 99% of latency values.

The average latency tells users how long to expect calls to take when considering a large number of calls. The 95th and 99th percentile latency numbers provide information about how much call times vary in cases where calls took longer than the average amount of time to complete. 95% of calls completed within the time specified by the 95th percentile number; 5% of calls took at least that long to complete. 99% of calls completed within the time specified by the 99th percentile number; 1% of calls took at least that long to complete.

For example, consider the following latency values:

Avg: 1 ms
95th: 3 ms
99th: 10 ms

If these were the measurements for 1000 calls to the store, then the average means that, overall, the 1000 calls took a total of 1000 ms (1000 x 1 ms), with a mix of call times, some less than 1 ms and some greater. The 95% and 99% values give some sense of how the call times varied over the set of calls. A 95% value of 3 ms means that, out of 1000 calls, 950 (95% of 1000) took less than 3 ms, and 50 (5% of 1000) took 3 ms or longer. A 99% value of 10 ms means that, out of 1000 calls, 990 (99% of 1000) took less than 10 ms, and 10 (1% of 1000) took 10 ms or longer.

The MultiGets_Interval_TotalOps stats tells how many records were read through MultiGets operations in the last interval. MultiGets_Cumulative_TotalOps stats tells how many records were read through MultiGets operations in the whole Replication Node lifetime.

A sample operation performance metrics:

{
  "resource": "rg1-rn1",
  "shard": "rg1",
  "reportTime": 1481031260001,
  "AllSingleKeyOperations_Interval_TotalOps": 154571,
  "AllSingleKeyOperations_Interval_TotalReq": 154571,
  "AllSingleKeyOperations_Interval_PerSec": 7728,
  "AllSingleKeyOperations_Interval_Min": 0,
  "AllSingleKeyOperations_Interval_Max": 72,
  "AllSingleKeyOperations_Interval_Avg": 0.09015835076570511,
  "AllSingleKeyOperations_Interval_95th": 0,
  "AllSingleKeyOperations_Interval_99th": 0,
  "AllSingleKeyOperations_Cumulative_TotalOps": 27916089,
  "AllSingleKeyOperations_Cumulative_TotalReq": 27916089,
  "AllSingleKeyOperations_Cumulative_PerSec": 854,
  "AllSingleKeyOperations_Cumulative_Min": 0,
  "AllSingleKeyOperations_Cumulative_Max": 5124,
  "AllSingleKeyOperations_Cumulative_Avg": 0.1090782955288887,
  "AllSingleKeyOperations_Cumulative_95th": 0,
  "AllSingleKeyOperations_Cumulative_99th": 0,
  "AllMultiKeyOperations_Interval_TotalOps": 6002,
  "AllMultiKeyOperations_Interval_TotalReq": 6002,
  "AllMultiKeyOperations_Interval_PerSec": 300,
  "AllMultiKeyOperations_Interval_Min": 0,
  "AllMultiKeyOperations_Interval_Max": 29,
  "AllMultiKeyOperations_Interval_Avg": 0.14758114516735077,
  "AllMultiKeyOperations_Interval_95th": 0,
  "AllMultiKeyOperations_Interval_99th": 1,
  "AllMultiKeyOperations_Cumulative_TotalOps": 1105133,
  "AllMultiKeyOperations_Cumulative_TotalReq": 1105133,
  "AllMultiKeyOperations_Cumulative_PerSec": 33,
  "AllMultiKeyOperations_Cumulative_Min": 0,
  "AllMultiKeyOperations_Cumulative_Max": 956,
  "AllMultiKeyOperations_Cumulative_Avg": 0.16301529109477997,
  "AllMultiKeyOperations_Cumulative_95th": 0,
  "AllMultiKeyOperations_Cumulative_99th": 1,
  "Gets_Interval_TotalOps": 154571,
  "Gets_Interval_TotalReq": 154571,
  "Gets_Interval_PerSec": 7728,
  "Gets_Interval_Min": 0,
  "Gets_Interval_Max": 72,
  "Gets_Interval_Avg": 0.08909573405981064,
  "Gets_Interval_95th": 0,
  "Gets_Interval_99th": 0,
  "Gets_Cumulative_TotalOps": 27916089,
  "Gets_Cumulative_TotalReq": 27916089,
  "Gets_Cumulative_PerSec": 854,
  "Gets_Cumulative_Min": 0,
  "Gets_Cumulative_Max": 5124,
  "Gets_Cumulative_Avg": 0.10803056508302689,
  "Gets_Cumulative_95th": 0,
  "Gets_Cumulative_99th": 0,
  "Puts_Interval_TotalOps": 0,
  "Puts_Interval_TotalReq": 0,
  "Puts_Interval_PerSec": 0,
  "Puts_Interval_Min": 0,
  "Puts_Interval_Max": 0,
  "Puts_Interval_Avg": 0,
  "Puts_Interval_95th": 0,
  "Puts_Interval_99th": 0,
  "PutIfAbsent_Interval_TotalOps": 0,
  "PutIfAbsent_Interval_TotalReq": 0,
  "PutIfAbsent_Interval_PerSec": 0,
  "PutIfAbsent_Interval_Min": 0,
  "PutIfAbsent_Interval_Max": 0,
  "PutIfAbsent_Interval_Avg": 0,
  "PutIfAbsent_Interval_95th": 0,
  "PutIfAbsent_Interval_99th": 0,
  "PutIfPresent_Interval_TotalOps": 0,
  "PutIfPresent_Interval_TotalReq": 0,
  "PutIfPresent_Interval_PerSec": 0,
  "PutIfPresent_Interval_Min": 0,
  "PutIfPresent_Interval_Max": 0,
  "PutIfPresent_Interval_Avg": 0,
  "PutIfPresent_Interval_95th": 0,
  "PutIfPresent_Interval_99th": 0,
  "PutIfVersion_Interval_TotalOps": 0,
  "PutIfVersion_Interval_TotalReq": 0,
  "PutIfVersion_Interval_PerSec": 0,
  "PutIfVersion_Interval_Min": 0,
  "PutIfVersion_Interval_Max": 0,
  "PutIfVersion_Interval_Avg": 0,
  "PutIfVersion_Interval_95th": 0,
  "PutIfVersion_Interval_99th": 0,
  "Puts_Cumulative_TotalOps": 0,
  "Puts_Cumulative_TotalReq": 0,
  "Puts_Cumulative_PerSec": 0,
  "Puts_Cumulative_Min": 0,
  "Puts_Cumulative_Max": 0,
  "Puts_Cumulative_Avg": 0,
  "Puts_Cumulative_95th": 0,
  "Puts_Cumulative_99th": 0,
  "PutIfAbsent_Cumulative_TotalOps": 0,
  "PutIfAbsent_Cumulative_TotalReq": 0,
  "PutIfAbsent_Cumulative_PerSec": 0,
  "PutIfAbsent_Cumulative_Min": 0,
  "PutIfAbsent_Cumulative_Max": 0,
  "PutIfAbsent_Cumulative_Avg": 0,
  "PutIfAbsent_Cumulative_95th": 0,
  "PutIfAbsent_Cumulative_99th": 0,
  "PutIfPresent_Cumulative_TotalOps": 0,
  "PutIfPresent_Cumulative_TotalReq": 0,
  "PutIfPresent_Cumulative_PerSec": 0,
  "PutIfPresent_Cumulative_Min": 0,
  "PutIfPresent_Cumulative_Max": 0,
  "PutIfPresent_Cumulative_Avg": 0,
  "PutIfPresent_Cumulative_95th": 0,
  "PutIfPresent_Cumulative_99th": 0,
  "PutIfVersion_Cumulative_TotalOps": 0,
  "PutIfVersion_Cumulative_TotalReq": 0,
  "PutIfVersion_Cumulative_PerSec": 0,
  "PutIfVersion_Cumulative_Min": 0,
  "PutIfVersion_Cumulative_Max": 0,
  "PutIfVersion_Cumulative_Avg": 0,
  "PutIfVersion_Cumulative_95th": 0,
  "PutIfVersion_Cumulative_99th": 0,
  "Deletes_Interval_TotalOps": 0,
  "Deletes_Interval_TotalReq": 0,
  "Deletes_Interval_PerSec": 0,
  "Deletes_Interval_Min": 0,
  "Deletes_Interval_Max": 0,
  "Deletes_Interval_Avg": 0,
  "Deletes_Interval_95th": 0,
  "Deletes_Interval_99th": 0,
  "DeleteIfVersion_Interval_TotalOps": 0,
  "DeleteIfVersion_Interval_TotalReq": 0,
  "DeleteIfVersion_Interval_PerSec": 0,
  "DeleteIfVersion_Interval_Min": 0,
  "DeleteIfVersion_Interval_Max": 0,
  "DeleteIfVersion_Interval_Avg": 0,
  "DeleteIfVersion_Interval_95th": 0,
  "DeleteIfVersion_Interval_99th": 0,
  "Deletes_Cumulative_TotalOps": 0,
  "Deletes_Cumulative_TotalReq": 0,
  "Deletes_Cumulative_PerSec": 0,
  "Deletes_Cumulative_Min": 0,
  "Deletes_Cumulative_Max": 0,
  "Deletes_Cumulative_Avg": 0,
  "Deletes_Cumulative_95th": 0,
  "Deletes_Cumulative_99th": 0,
  "DeleteIfVersion_Cumulative_TotalOps": 0,
  "DeleteIfVersion_Cumulative_TotalReq": 0,
  "DeleteIfVersion_Cumulative_PerSec": 0,
  "DeleteIfVersion_Cumulative_Min": 0,
  "DeleteIfVersion_Cumulative_Max": 0,
  "DeleteIfVersion_Cumulative_Avg": 0,
  "DeleteIfVersion_Cumulative_95th": 0,
  "DeleteIfVersion_Cumulative_99th": 0,
  "MultiGets_Interval_TotalOps": 0,
  "MultiGets_Interval_TotalReq": 0,
  "MultiGets_Interval_PerSec": 0,
  "MultiGets_Interval_Min": 0,
  "MultiGets_Interval_Max": 0,
  "MultiGets_Interval_Avg": 0,
  "MultiGets_Interval_95th": 0,
  "MultiGets_Interval_99th": 0,
  "MultiGetKeys_Interval_TotalOps": 0,
  "MultiGetKeys_Interval_TotalReq": 0,
  "MultiGetKeys_Interval_PerSec": 0,
  "MultiGetKeys_Interval_Min": 0,
  "MultiGetKeys_Interval_Max": 0,
  "MultiGetKeys_Interval_Avg": 0,
  "MultiGetKeys_Interval_95th": 0,
  "MultiGetKeys_Interval_99th": 0,
  "MultiGetIterator_Interval_TotalOps": 0,
  "MultiGetIterator_Interval_TotalReq": 0,
  "MultiGetIterator_Interval_PerSec": 0,
  "MultiGetIterator_Interval_Min": 0,
  "MultiGetIterator_Interval_Max": 0,
  "MultiGetIterator_Interval_Avg": 0,
  "MultiGetIterator_Interval_95th": 0,
  "MultiGetIterator_Interval_99th": 0,
  "MultiGetKeysIterator_Interval_TotalOps": 0,
  "MultiGetKeysIterator_Interval_TotalReq": 0,
  "MultiGetKeysIterator_Interval_PerSec": 0,
  "MultiGetKeysIterator_Interval_Min": 0,
  "MultiGetKeysIterator_Interval_Max": 0,
  "MultiGetKeysIterator_Interval_Avg": 0,
  "MultiGetKeysIterator_Interval_95th": 0,
  "MultiGetKeysIterator_Interval_99th": 0,
  "MultiGets_Cumulative_TotalOps": 0,
  "MultiGets_Cumulative_TotalReq": 0,
  "MultiGets_Cumulative_PerSec": 0,
  "MultiGets_Cumulative_Min": 0,
  "MultiGets_Cumulative_Max": 0,
  "MultiGets_Cumulative_Avg": 0,
  "MultiGets_Cumulative_95th": 0,
  "MultiGets_Cumulative_99th": 0,
  "MultiGetKeys_Cumulative_TotalOps": 0,
  "MultiGetKeys_Cumulative_TotalReq": 0,
  "MultiGetKeys_Cumulative_PerSec": 0,
  "MultiGetKeys_Cumulative_Min": 0,
  "MultiGetKeys_Cumulative_Max": 0,
  "MultiGetKeys_Cumulative_Avg": 0,
  "MultiGetKeys_Cumulative_95th": 0,
  "MultiGetKeys_Cumulative_99th": 0,
  "MultiGetIterator_Cumulative_TotalOps": 0,
  "MultiGetIterator_Cumulative_TotalReq": 0,
  "MultiGetIterator_Cumulative_PerSec": 0,
  "MultiGetIterator_Cumulative_Min": 0,
  "MultiGetIterator_Cumulative_Max": 0,
  "MultiGetIterator_Cumulative_Avg": 0,
  "MultiGetIterator_Cumulative_95th": 0,
  "MultiGetIterator_Cumulative_99th": 0,
  "MultiGetKeysIterator_Cumulative_TotalOps": 0,
  "MultiGetKeysIterator_Cumulative_TotalReq": 0,
  "MultiGetKeysIterator_Cumulative_PerSec": 0,
  "MultiGetKeysIterator_Cumulative_Min": 0,
  "MultiGetKeysIterator_Cumulative_Max": 0,
  "MultiGetKeysIterator_Cumulative_Avg": 0,
  "MultiGetKeysIterator_Cumulative_95th": 0,
  "MultiGetKeysIterator_Cumulative_99th": 0,
  "StoreIterator_Interval_TotalOps": 0,
  "StoreIterator_Interval_TotalReq": 0,
  "StoreIterator_Interval_PerSec": 0,
  "StoreIterator_Interval_Min": 0,
  "StoreIterator_Interval_Max": 0,
  "StoreIterator_Interval_Avg": 0,
  "StoreIterator_Interval_95th": 0,
  "StoreIterator_Interval_99th": 0,
  "StoreKeysIterator_Interval_TotalOps": 0,
  "StoreKeysIterator_Interval_TotalReq": 0,
  "StoreKeysIterator_Interval_PerSec": 0,
  "StoreKeysIterator_Interval_Min": 0,
  "StoreKeysIterator_Interval_Max": 0,
  "StoreKeysIterator_Interval_Avg": 0,
  "StoreKeysIterator_Interval_95th": 0,
  "StoreKeysIterator_Interval_99th": 0,
  "StoreIterator_Cumulative_TotalOps": 0,
  "StoreIterator_Cumulative_TotalReq": 0,
  "StoreIterator_Cumulative_PerSec": 0,
  "StoreIterator_Cumulative_Min": 0,
  "StoreIterator_Cumulative_Max": 0,
  "StoreIterator_Cumulative_Avg": 0,
  "StoreIterator_Cumulative_95th": 0,
  "StoreIterator_Cumulative_99th": 0,
  "StoreKeysIterator_Cumulative_TotalOps": 0,
  "StoreKeysIterator_Cumulative_TotalReq": 0,
  "StoreKeysIterator_Cumulative_PerSec": 0,
  "StoreKeysIterator_Cumulative_Min": 0,
  "StoreKeysIterator_Cumulative_Max": 0,
  "StoreKeysIterator_Cumulative_Avg": 0,
  "StoreKeysIterator_Cumulative_95th": 0,
  "StoreKeysIterator_Cumulative_99th": 0,
  "MultiDeletes_Interval_TotalOps": 0,
  "MultiDeletes_Interval_TotalReq": 0,
  "MultiDeletes_Interval_PerSec": 0,
  "MultiDeletes_Interval_Min": 0,
  "MultiDeletes_Interval_Max": 0,
  "MultiDeletes_Interval_Avg": 0,
  "MultiDeletes_Interval_95th": 0,
  "MultiDeletes_Interval_99th": 0,
  "MultiDeletes_Cumulative_TotalOps": 0,
  "MultiDeletes_Cumulative_TotalReq": 0,
  "MultiDeletes_Cumulative_PerSec": 0,
  "MultiDeletes_Cumulative_Min": 0,
  "MultiDeletes_Cumulative_Max": 0,
  "MultiDeletes_Cumulative_Avg": 0,
  "MultiDeletes_Cumulative_95th": 0,
  "MultiDeletes_Cumulative_99th": 0,
  "Executes_Interval_TotalOps": 0,
  "Executes_Interval_TotalReq": 0,
  "Executes_Interval_PerSec": 0,
  "Executes_Interval_Min": 0,
  "Executes_Interval_Max": 0,
  "Executes_Interval_Avg": 0,
  "Executes_Interval_95th": 0,
  "Executes_Interval_99th": 0,
  "Executes_Cumulative_TotalOps": 0,
  "Executes_Cumulative_TotalReq": 0,
  "Executes_Cumulative_PerSec": 0,
  "Executes_Cumulative_Min": 0,
  "Executes_Cumulative_Max": 0,
  "Executes_Cumulative_Avg": 0,
  "Executes_Cumulative_95th": 0,
  "Executes_Cumulative_99th": 0,
  "NOPs_Interval_TotalOps": 0,
  "NOPs_Interval_TotalReq": 0,
  "NOPs_Interval_PerSec": 0,
  "NOPs_Interval_Min": 0,
  "NOPs_Interval_Max": 0,
  "NOPs_Interval_Avg": 0,
  "NOPs_Interval_95th": 0,
  "NOPs_Interval_99th": 0,
  "NOPs_Cumulative_TotalOps": 0,
  "NOPs_Cumulative_TotalReq": 0,
  "NOPs_Cumulative_PerSec": 0,
  "NOPs_Cumulative_Min": 0,
  "NOPs_Cumulative_Max": 0,
  "NOPs_Cumulative_Avg": 0,
  "NOPs_Cumulative_95th": 0,
  "NOPs_Cumulative_99th": 0,
  "IndexIterator_Interval_TotalOps": 6002,
  "IndexIterator_Interval_TotalReq": 6002,
  "IndexIterator_Interval_PerSec": 300,
  "IndexIterator_Interval_Min": 0,
  "IndexIterator_Interval_Max": 29,
  "IndexIterator_Interval_Avg": 0.14662425220012665,
  "IndexIterator_Interval_95th": 0,
  "IndexIterator_Interval_99th": 1,
  "IndexKeysIterator_Interval_TotalOps": 0,
  "IndexKeysIterator_Interval_TotalReq": 0,
  "IndexKeysIterator_Interval_PerSec": 0,
  "IndexKeysIterator_Interval_Min": 0,
  "IndexKeysIterator_Interval_Max": 0,
  "IndexKeysIterator_Interval_Avg": 0,
  "IndexKeysIterator_Interval_95th": 0,
  "IndexKeysIterator_Interval_99th": 0,
  "IndexIterator_Cumulative_TotalOps": 1105133,
  "IndexIterator_Cumulative_TotalReq": 1105133,
  "IndexIterator_Cumulative_PerSec": 33,
  "IndexIterator_Cumulative_Min": 0,
  "IndexIterator_Cumulative_Max": 956,
  "IndexIterator_Cumulative_Avg": 0.1620502769947052,
  "IndexIterator_Cumulative_95th": 0,
  "IndexIterator_Cumulative_99th": 1,
  "IndexKeysIterator_Cumulative_TotalOps": 0,
  "IndexKeysIterator_Cumulative_TotalReq": 0,
  "IndexKeysIterator_Cumulative_PerSec": 0,
  "IndexKeysIterator_Cumulative_Min": 0,
  "IndexKeysIterator_Cumulative_Max": 0,
  "IndexKeysIterator_Cumulative_Avg": 0,
  "IndexKeysIterator_Cumulative_95th": 0,
  "IndexKeysIterator_Cumulative_99th": 0,
  "QuerySinglePartition_Interval_TotalOps": 0,
  "QuerySinglePartition_Interval_TotalReq": 0,
  "QuerySinglePartition_Interval_PerSec": 0,
  "QuerySinglePartition_Interval_Min": 0,
  "QuerySinglePartition_Interval_Max": 0,
  "QuerySinglePartition_Interval_Avg": 0,
  "QuerySinglePartition_Interval_95th": 0,
  "QuerySinglePartition_Interval_99th": 0,
  "QueryMultiPartition_Interval_TotalOps": 0,
  "QueryMultiPartition_Interval_TotalReq": 0,
  "QueryMultiPartition_Interval_PerSec": 0,
  "QueryMultiPartition_Interval_Min": 0,
  "QueryMultiPartition_Interval_Max": 0,
  "QueryMultiPartition_Interval_Avg": 0,
  "QueryMultiPartition_Interval_95th": 0,
  "QueryMultiPartition_Interval_99th": 0,
  "QueryMultiShard_Interval_TotalOps": 0,
  "QueryMultiShard_Interval_TotalReq": 0,
  "QueryMultiShard_Interval_PerSec": 0,
  "QueryMultiShard_Interval_Min": 0,
  "QueryMultiShard_Interval_Max": 0,
  "QueryMultiShard_Interval_Avg": 0,
  "QueryMultiShard_Interval_95th": 0,
  "QueryMultiShard_Interval_99th": 0,
  "QuerySinglePartition_Cumulative_TotalOps": 0,
  "QuerySinglePartition_Cumulative_TotalReq": 0,
  "QuerySinglePartition_Cumulative_PerSec": 0,
  "QuerySinglePartition_Cumulative_Min": 0,
  "QuerySinglePartition_Cumulative_Max": 0,
  "QuerySinglePartition_Cumulative_Avg": 0,
  "QuerySinglePartition_Cumulative_95th": 0,
  "QuerySinglePartition_Cumulative_99th": 0,
  "QueryMultiPartition_Cumulative_TotalOps": 0,
  "QueryMultiPartition_Cumulative_TotalReq": 0,
  "QueryMultiPartition_Cumulative_PerSec": 0,
  "QueryMultiPartition_Cumulative_Min": 0,
  "QueryMultiPartition_Cumulative_Max": 0,
  "QueryMultiPartition_Cumulative_Avg": 0,
  "QueryMultiPartition_Cumulative_95th": 0,
  "QueryMultiPartition_Cumulative_99th": 0,
  "QueryMultiShard_Cumulative_TotalOps": 0,
  "QueryMultiShard_Cumulative_TotalReq": 0,
  "QueryMultiShard_Cumulative_PerSec": 0,
  "QueryMultiShard_Cumulative_Min": 0,
  "QueryMultiShard_Cumulative_Max": 0,
  "QueryMultiShard_Cumulative_Avg": 0,
  "QueryMultiShard_Cumulative_95th": 0,
  "QueryMultiShard_Cumulative_99th": 0,
  "BulkPut_Interval_TotalOps": 0,
  "BulkPut_Interval_TotalReq": 0,
  "BulkPut_Interval_PerSec": 0,
  "BulkPut_Interval_Min": 0,
  "BulkPut_Interval_Max": 0,
  "BulkPut_Interval_Avg": 0,
  "BulkPut_Interval_95th": 0,
  "BulkPut_Interval_99th": 0,
  "BulkPut_Cumulative_TotalOps": 0,
  "BulkPut_Cumulative_TotalReq": 0,
  "BulkPut_Cumulative_PerSec": 0,
  "BulkPut_Cumulative_Min": 0,
  "BulkPut_Cumulative_Max": 0,
  "BulkPut_Cumulative_Avg": 0,
  "BulkPut_Cumulative_95th": 0,
  "BulkPut_Cumulative_99th": 0,
  "BulkGet_Interval_TotalOps": 0,
  "BulkGet_Interval_TotalReq": 0,
  "BulkGet_Interval_PerSec": 0,
  "BulkGet_Interval_Min": 0,
  "BulkGet_Interval_Max": 0,
  "BulkGet_Interval_Avg": 0,
  "BulkGet_Interval_95th": 0,
  "BulkGet_Interval_99th": 0,
  "BulkGetKeys_Interval_TotalOps": 0,
  "BulkGetKeys_Interval_TotalReq": 0,
  "BulkGetKeys_Interval_PerSec": 0,
  "BulkGetKeys_Interval_Min": 0,
  "BulkGetKeys_Interval_Max": 0,
  "BulkGetKeys_Interval_Avg": 0,
  "BulkGetKeys_Interval_95th": 0,
  "BulkGetKeys_Interval_99th": 0,
  "BulkGetTable_Interval_TotalOps": 0,
  "BulkGetTable_Interval_TotalReq": 0,
  "BulkGetTable_Interval_PerSec": 0,
  "BulkGetTable_Interval_Min": 0,
  "BulkGetTable_Interval_Max": 0,
  "BulkGetTable_Interval_Avg": 0,
  "BulkGetTable_Interval_95th": 0,
  "BulkGetTable_Interval_99th": 0,
  "BulkGetTableKeys_Interval_TotalOps": 0,
  "BulkGetTableKeys_Interval_TotalReq": 0,
  "BulkGetTableKeys_Interval_PerSec": 0,
  "BulkGetTableKeys_Interval_Min": 0,
  "BulkGetTableKeys_Interval_Max": 0,
  "BulkGetTableKeys_Interval_Avg": 0,
  "BulkGetTableKeys_Interval_95th": 0,
  "BulkGetTableKeys_Interval_99th": 0,
  "BulkGet_Cumulative_TotalOps": 0,
  "BulkGet_Cumulative_TotalReq": 0,
  "BulkGet_Cumulative_PerSec": 0,
  "BulkGet_Cumulative_Min": 0,
  "BulkGet_Cumulative_Max": 0,
  "BulkGet_Cumulative_Avg": 0,
  "BulkGet_Cumulative_95th": 0,
  "BulkGet_Cumulative_99th": 0,
  "BulkGetKeys_Cumulative_TotalOps": 0,
  "BulkGetKeys_Cumulative_TotalReq": 0,
  "BulkGetKeys_Cumulative_PerSec": 0,
  "BulkGetKeys_Cumulative_Min": 0,
  "BulkGetKeys_Cumulative_Max": 0,
  "BulkGetKeys_Cumulative_Avg": 0,
  "BulkGetKeys_Cumulative_95th": 0,
  "BulkGetKeys_Cumulative_99th": 0,
  "BulkGetTable_Cumulative_TotalOps": 0,
  "BulkGetTable_Cumulative_TotalReq": 0,
  "BulkGetTable_Cumulative_PerSec": 0,
  "BulkGetTable_Cumulative_Min": 0,
  "BulkGetTable_Cumulative_Max": 0,
  "BulkGetTable_Cumulative_Avg": 0,
  "BulkGetTable_Cumulative_95th": 0,
  "BulkGetTable_Cumulative_99th": 0,
  "BulkGetTableKeys_Cumulative_TotalOps": 0,
  "BulkGetTableKeys_Cumulative_TotalReq": 0,
  "BulkGetTableKeys_Cumulative_PerSec": 0,
  "BulkGetTableKeys_Cumulative_Min": 0,
  "BulkGetTableKeys_Cumulative_Max": 0,
  "BulkGetTableKeys_Cumulative_Avg": 0,
  "BulkGetTableKeys_Cumulative_95th": 0,
  "BulkGetTableKeys_Cumulative_99th": 0
}

New detailed statistics of single environment and replicated environment are available as follows:

Type: oracle.kv.repnode.envmetric
User Data: contains a full listing of detailed statistics for a given RN. The statistics are a string in JSON form, and are obtained through Notification.getUserData(). See the javadoc for EnvironmentStats and ReplicatedEnvironmentStats for more information about the meaning of the statistics.

An example stat is: getReplicaVLSNLagMap() – Returns a map from replica node name to the lag, in VLSNs, between the replication state of the replica and the master, if known. Returns an empty map if this node is not the master.

A sample statistics of single environment and replicated environment:

{
  "resource": "rg1-rn1",
  "shard": "rg1",
  "reportTime": 1498021100001,
  "FeederManager_nMaxReplicaLag": -1,
  "FeederManager_replicaLastCommitTimestampMap":
  "rg1-rn2=1498021098996;rg1-rn3=1498021096989",
  "FeederManager_nFeedersShutdown": 0,
  "FeederManager_nFeedersCreated": 2,
  "FeederManager_nMaxReplicaLagName": "rg1-rn2",
  "FeederManager_replicaVLSNLagMap": "rg1-rn2=0;rg1-rn3=2",
  "FeederManager_replicaVLSNRateMap": "rg1-rn2=472;rg1-rn3=472",
  "FeederManager_replicaDelayMap": "rg1-rn2=0;rg1-rn3=2007",
  "FeederManager_replicaLastCommitVLSNMap": "rg1-rn2=836;rg1-rn3=834",
  "FeederTxns_txnsAcked": 77,
  "FeederTxns_lastCommitVLSN": 848,
  "FeederTxns_totalTxnMS": 228,
  "FeederTxns_lastCommitTimestamp": 1498021099030,
  "FeederTxns_vlsnRate": 439,
  "FeederTxns_txnsNotAcked": 0,
  "FeederTxns_ackWaitMS": 115,
  "Replay_nAborts": 0,
  "Replay_nGroupCommits": 0,
  "Replay_nNameLNs": 0,
  "Replay_nElapsedTxnTime": 0,
  "Replay_nMessageQueueOverflows": 0,
  "Replay_nGroupCommitMaxExceeded": 0,
  "Replay_nCommitSyncs": 0,
  "Replay_nCommitNoSyncs": 0,
  "Replay_maxCommitProcessingNanos": 0,
  "Replay_nGroupCommitTxns": 0,
  "Replay_nCommitWriteNoSyncs": 0,
  "Replay_minCommitProcessingNanos": 0,
  "Replay_nCommitAcks": 0,
  "Replay_nLNs": 0,
  "Replay_nCommits": 0,
  "Replay_latestCommitLagMs": 0,
  "Replay_totalCommitLagMs": 0,
  "Replay_totalCommitProcessingNanos": 0,
  "Replay_nGroupCommitTimeouts": 0,
  "ConsistencyTracker_nVLSNConsistencyWaitMS": 0,
  "ConsistencyTracker_nLagConsistencyWaits": 0,
  "ConsistencyTracker_nLagConsistencyWaitMS": 0,
  "ConsistencyTracker_nVLSNConsistencyWaits": 0,
  "BinaryProtocol_nMaxGroupedAcks": 0,
  "BinaryProtocol_messagesWrittenPerSecond": 19646,
  "BinaryProtocol_nEntriesOldVersion": 0,
  "BinaryProtocol_bytesReadPerSecond": 0,
  "BinaryProtocol_bytesWrittenPerSecond": 1057385,
  "BinaryProtocol_nMessagesWritten": 344,
  "BinaryProtocol_nGroupAckMessages": 0,
  "BinaryProtocol_nMessagesRead": 0,
  "BinaryProtocol_nReadNanos": 0,
  "BinaryProtocol_nMessageBatches": 24,
  "BinaryProtocol_nAckMessages": 0,
  "BinaryProtocol_nWriteNanos": 17509221,
  "BinaryProtocol_nBytesRead": 0,
  "BinaryProtocol_nGroupedAcks": 0,
  "BinaryProtocol_nMessagesBatched": 48,
  "BinaryProtocol_messagesReadPerSecond": 0,
  "BinaryProtocol_nBytesWritten": 18514,
  "VLSNIndex_nHeadBucketsDeleted": 0,
  "VLSNIndex_nBucketsCreated": 0,
  "VLSNIndex_nMisses": 0,
  "VLSNIndex_nTailBucketsDeleted": 0,
  "VLSNIndex_nHits": 19,
  "I/O_nRepeatFaultReads": 0,
  "I/O_nRandomReads": 0,
  "I/O_nLogIntervalExceeded": 0,
  "I/O_nTempBufferWrites": 0,
  "I/O_nWriteQueueOverflowFailures": 0,
  "I/O_nGroupCommitWaits": 0,
  "I/O_nGroupCommitRequests": 0,
  "I/O_nWritesFromWriteQueue": 0,
  "I/O_nSequentialWrites": 4,
  "I/O_nGrpCommitTimeouts": 0,
  "I/O_nFileOpens": 0,
  "I/O_nRandomWrites": 0,
  "I/O_bufferBytes": 4404016,
  "I/O_nSequentialReadBytes": 0,
  "I/O_endOfLog": 135573,
  "I/O_nSequentialWriteBytes": 16693,
  "I/O_nFSyncTime": 114,
  "I/O_nSequentialReads": 0,
  "I/O_nLogFSyncs": 1,
  "I/O_nNoFreeBuffer": 0,
  "I/O_nFSyncs": 0,
  "I/O_nCacheMiss": 0,
  "I/O_nWriteQueueOverflow": 0,
  "I/O_nRandomWriteBytes": 0,
  "I/O_nReadsFromWriteQueue": 0,
  "I/O_nBytesReadFromWriteQueue": 0,
  "I/O_nBytesWrittenFromWriteQueue": 0,
  "I/O_nNotResident": 21,
  "I/O_nFSyncRequests": 0,
  "I/O_nRandomReadBytes": 0,
  "I/O_nOpenFiles": 1,
  "I/O_nLogBuffers": 16,
  "I/O_nLogMaxGroupCommitThreshold": 0,
  "I/O_nFSyncMaxTime": 114,
  "Cache_nBytesEvictedCACHEMODE": 0,
  "Cache_nINSparseTarget": 85,
  "Cache_nINNoTarget": 81,
  "Cache_dataAdminBytes": 48800,
  "Cache_nBINsFetchMiss": 0,
  "Cache_nNodesEvicted": 0,
  "Cache_cacheTotalBytes": 5125248,
  "Cache_nSharedCacheEnvironments": 0,
  "Cache_nEvictionRuns": 0,
  "Cache_lruMixedSize": 90,
  "Cache_nLNsEvicted": 0,
  "Cache_nBINsFetch": 92,
  "Cache_nNodesMovedToDirtyLRU": 0,
  "Cache_nLNsFetch": 1761,
  "Cache_nBytesEvictedDAEMON": 0,
  "Cache_nDirtyNodesEvicted": 0,
  "Cache_nUpperINsFetchMiss": 0,
  "Cache_nCachedBINs": 90,
  "Cache_nNodesMutated": 0,
  "Cache_nNodesStripped": 0,
  "Cache_dataBytes": 686704,
  "Cache_nFullBINsMiss": 0,
  "Cache_nBINsFetchMissRatio": 0,
  "Cache_nRootNodesEvicted": 0,
  "Cache_nCachedBINDeltas": 0,
  "Cache_nBytesEvictedMANUAL": 0,
  "Cache_nNodesSkipped": 0,
  "Cache_nUpperINsFetch": 0,
  "Cache_nBinDeltaBlindOps": 0,
  "Cache_nBytesEvictedCRITICAL": 0,
  "Cache_lruDirtySize": 0,
  "Cache_nLNsFetchMiss": 21,
  "Cache_adminBytes": 589,
  "Cache_nBINDeltasFetchMiss": 0,
  "Cache_nThreadUnavailable": 0,
  "Cache_nCachedUpperINs": 84,
  "Cache_sharedCacheTotalBytes": 0,
  "Cache_nNodesPutBack": 0,
  "Cache_nBytesEvictedEVICTORTHREAD": 0,
  "Cache_DOSBytes": 0,
  "Cache_lockBytes": 33936,
  "Cache_nNodesTargeted": 0,
  "Cache_nINCompactKey": 7,
  "OffHeap_offHeapCriticalNodesTargeted": 0,
  "OffHeap_offHeapDirtyNodesEvicted": 0,
  "OffHeap_offHeapNodesSkipped": 4,
  "OffHeap_offHeapLNsEvicted": 44,
  "OffHeap_offHeapAllocOverflow": 0,
  "OffHeap_offHeapCachedLNs": 0,
  "OffHeap_offHeapNodesStripped": 44,
  "OffHeap_offHeapLruSize": 0,
  "OffHeap_offHeapLNsStored": 44,
  "OffHeap_offHeapLNsLoaded": 22,
  "OffHeap_offHeapTotalBytes": 0,
  "OffHeap_offHeapTotalBlocks": 0,
  "OffHeap_offHeapNodesEvicted": 0,
  "OffHeap_offHeapCachedBINDeltas": 0,
  "OffHeap_offHeapNodesMutated": 0,
  "OffHeap_offHeapNodesTargeted": 48,
  "OffHeap_offHeapCachedBINs": 0,
  "OffHeap_offHeapAllocFailure": 0,
  "OffHeap_offHeapBINsLoaded": 0,
  "OffHeap_offHeapThreadUnavailable": 63,
  "OffHeap_offHeapBINsStored": 0,
  "Cleaning_nBINDeltasMigrated": 0,
  "Cleaning_minUtilization": 68,
  "Cleaning_nLNsMigrated": 0,
  "Cleaning_nINsCleaned": 0,
  "Cleaning_nPendingLNsProcessed": 0,
  "Cleaning_nToBeCleanedLNsProcessed": 0,
  "Cleaning_nLNsLocked": 0,
  "Cleaning_nRevisalRuns": 0,
  "Cleaning_nPendingLNsLocked": 0,
  "Cleaning_nTwoPassRuns": 0,
  "Cleaning_nBINDeltasObsolete": 0,
  "Cleaning_maxUtilization": 68,
  "Cleaning_nLNsMarked": 0,
  "Cleaning_pendingLNQueueSize": 0,
  "Cleaning_nMarkLNsProcessed": 0,
  "Cleaning_nRepeatIteratorReads": 0,
  "Cleaning_nLNsExpired": 0,
  "Cleaning_nCleanerRuns": 0,
  "Cleaning_nBINDeltasDead": 0,
  "Cleaning_nCleanerDisksReads": 0,
  "Cleaning_protectedLogSizeMap": "",
  "Cleaning_nCleanerDeletions": 0,
  "Cleaning_nCleanerEntriesRead": 0,
  "Cleaning_availableLogSize": 48942137344,
  "Cleaning_nLNsDead": 0,
  "Cleaning_nINsObsolete": 0,
  "Cleaning_activeLogSize": 112716,
  "Cleaning_nINsDead": 0,
  "Cleaning_nINsMigrated": 0,
  "Cleaning_totalLogSize": 112716,
  "Cleaning_nBINDeltasCleaned": 0,
  "Cleaning_nLNsObsolete": 0,
  "Cleaning_nLNsCleaned": 0,
  "Cleaning_nLNQueueHits": 0,
  "Cleaning_reservedLogSize": 0,
  "Cleaning_protectedLogSize": 0,
  "Cleaning_nClusterLNsProcessed": 0,
  "Node Compression_processedBins": 0,
  "Node Compression_splitBins": 0,
  "Node Compression_dbClosedBins": 0,
  "Node Compression_cursorsBins": 0,
  "Node Compression_nonEmptyBins": 0,
  "Node Compression_inCompQueueSize": 0,
  "Checkpoints_lastCheckpointInterval": 670,
  "Checkpoints_nDeltaINFlush": 0,
  "Checkpoints_lastCheckpointStart": 670,
  "Checkpoints_lastCheckpointEnd": 1342,
  "Checkpoints_nFullBINFlush": 0,
  "Checkpoints_lastCheckpointId": 1,
  "Checkpoints_nFullINFlush": 0,
  "Checkpoints_nCheckpoints": 0,
  "Environment_nBinDeltaInsert": 0,
  "Environment_nBinDeltaUpdate": 0,
  "Environment_nBinDeltaGet": 0,
  "Environment_btreeRelatchesRequired": 0,
  "Environment_nBinDeltaDelete": 0,
  "Environment_environmentCreationTime": 1498021055255,
  "Locks_nWaiters": 0,
  "Locks_nRequests": 142,
  "Locks_nLatchAcquiresSelfOwned": 0,
  "Locks_nWriteLocks": 0,
  "Locks_nTotalLocks": 303,
  "Locks_nReadLocks": 303,
  "Locks_nLatchAcquiresNoWaitSuccessful": 0,
  "Locks_nOwners": 303,
  "Locks_nLatchAcquiresWithContention": 0,
  "Locks_nLatchAcquireNoWaitUnsuccessful": 0,
  "Locks_nLatchReleases": 0,
  "Locks_nLatchAcquiresNoWaiters": 0,
  "Locks_nWaits": 0,
  "Op_secSearchFail": 0,
  "Op_priDelete": 0,
  "Op_priSearchFail": 14,
  "Op_secPosition": 0,
  "Op_priInsertFail": 0,
  "Op_priDeleteFail": 0,
  "Op_secSearch": 0,
  "Op_priSearch": 54,
  "Op_priPosition": 2,
  "Op_secDelete": 0,
  "Op_secUpdate": 0,
  "Op_secInsert": 0,
  "Op_priUpdate": 11,
  "Op_priInsert": 66
}

Announce a change in this RepNode’s replication state.
- Type: oracle.kv.repnode.replicationstate
- User Data: RN replication state change event. The event is a string in JSON form, and is obtained via Notification.getUserData().
For example:
```
{"resource":"rg1-rn3","shard":"rg1","reportTime":1476980297641,
"replication_state":"MASTER"}
```
Announce a change in this RepNode’s service status.
- Type: oracle.kv.repnode.status
- User Data: RN service status change event. The event is a string in JSON form, and is obtained via Notification.getUserData().
For example:
```
{"resource":"rg3-rn3","shard":"rg3","reportTime":1476981010202,
"service_status":"ERROR_RESTARTING"}
```
Announce a plan state change.
- Type: oracle.kv.plan.status
- User Data: Plan status change event. The event is a string in JSON form, and is obtained via Notification.getUserData().
For example:
```
{"planId":7,"planName":"Change Global Params
(7)","reportTime":1477272558763,"state":"SUCCEEDED","attemptNumber":1,
"message":"Plan finished."}
```