Managing Off-heap Cache

Each Replication Node in the store divides the memory available to it between memory containing store objects, memory used by the host operating system (which includes the file system cache), and the in-memory cache used to contain frequently accessed store data. You can configure the amount of memory used by the cache using the rnHeapPercent parameter. The value for this parameter is expressed as a percentage of the total memory available to the Replication Node (RN). In turn, memory available to the RN is determined by the amount of memory available to the Java VM. By default, 70% of the memory available to the Replication Node is reserved for in-memory cache.

However, this can and probably will fill up. When that happens, objects are evicted from the cache based on a Least-Recently-Used (LRU) algorithm. The LRU algorithm guarantees that the most commonly accessed objects remain in the cache.

You can configure your Replication Node so that the in-memory cache is very large, and therefore, less likely to overflow. However, several drawbacks occur when the cache is too large. Most importantly, a large cache decreases the Java Garbage Collection performance. In turn, this adversely affects overall performance for your Replication Node. As a general good practice, keep the heap size to less than 32 GB. Doing so limits in-memory cache to about 22.4 GB.

Note:

Oracle NoSQL Database actually limits your maximum heap size to a default value of 32 GB to prevent you from inadvertently creating an excessively large cache. The value of the maximum heap size is managed with the rnHeapMaxMB parameter, described later in this section.

When you decrease the heap size of a Replication Node, you can configure the system so that any unused heap memory is used by the operating system and file system cache. A large file system cache has performance benefits, as well as drawbacks:

Significant redundancy exists between the main cache and file system cache because all data and Btree information written by the Replication Node appears in the file system and can also appear in the main cache.
It is not possible for dirty Btree information to be placed in the file system cache without logging it. Such logging may be unnecessary, and will create additional work for internal cleaner threads.

To avoid problems caused by excessively large in-memory and file system caches, the default configuration for your Replication Node uses an off-heap cache. The off-heap cache holds record data and Btree nodes when they are evicted from the main cache because it overflows. When off-heap cache overflows, eviction occurs according to the same LRU algorithm used by the main cache.

If off-heap allocations (or others) fail because there is no available memory, the process is likely to die without throwing an exception. In one test on Linux, for example, the process was killed abruptly by the OS. The only indication of the problem was the following shown by dmesg.

 Out of memory: Kill process 28768 (java) score 974 or sacrifice child
 Killed process 28768 (java)
    total-vm:278255336kB, anon-rss:257274420kB, file-rss:0kB

Note:

Before modifying the caches in use by your production store, always complete performance tests first.

Configuring the Off-heap Cache

You do not directly control the amount of memory available to the off-heap cache. Instead, you make memory available to it mainly by limiting the amount of memory available to the operating system. You might also need to control the size of the Java heap, which in turn controls the size of the in-memory cache. Any Storage Node memory that remains once the heap, JVM overhead, and operating system requirements are met is used for the off-heap cache. Any Replication Node memory that remains once the heap and operating system requirements are met is used for the off-heap cache. If no memory remains, the off-heap cache is not used.

The parameters you use to (indirectly) configure the off-heap cache are:

systemPercent – Defines the percentage of memory available to the Storage Node that is available for operating system use after the heap requirement is subtracted. By default, this value is 10%. Configuring this value to a number that is greater than 10% but less than 100% might leave room for the off-heap cache (depending on the memory on your Storage Node and the value chosen for this parameter). If room is available for the off-heap cache, it will be turned on.
For most production systems, if you are using the off-heap cache, a default value of 10% should be sufficient.
rnHeapMaxMB – The maximum amount of memory available for the Java heap. (The heap is where the in-memory cache is contained.) The size of your heap will be the lesser of this value, or the size arrived at using the rnHeapPercent parameter value:
```
<b>total SN memory</b> * rnHeapPercent = Heap Size 
```
The default rnHeapPercent size is 68%. So if your Storage Node has 32GB memory, then the heap size will be 32 * 0.68 = 21.76 GB. However, if you set your rnHeapMaxMB to 20,480 (20 * 1024), then your heap size will be 20GB instead.
Remember, that the heap size is not the same as the in-memory cache size. The in-memory cache size is expressed as a percentage of the heap size. By default, this is 70% (configurable using the rnCachePercent parameter). So, if your heap size is 21.76 GB, then your in-memory cache size ends up being 15.23 GB (21.76 * 0.7).

Note:
Regardless of the values provided to the parameters described here, and the actual memory available for the heap, your heap size is limited to a maximum size of 32 GB.
jvmOverheadPercent – Defines the percentage of the Java heap that is available for JVM overhead (outside the Java heap). By default, this value is 25%, and heap and JVM overhead would take away 85% (68 + 68 * 0.25) of the Storage Node memory.

For example, if your Storage Node has 80 GB of memory, and just one Replication Node, then by default:

The heap size is 32 GB (80 * .68 = 54.4, which is greater than the 32 GB maximum).
The in-memory cache size is 22.4 GB (32 * 0.7).
The JVM overhead memory size is 8 GB (32 * 0.25).
A system memory size of 8 GB (80 * 0.1).
An off-heap cache size of 32 GB. This is the amount of memory that remains after the heap, JVM overhead, and system requirements have been met.

Figure E-1 SN Memory With Off-heap Cache

If you want to eliminate the off-heap cache, then set system memory to 100%. Do this for each Storage Node in your store. Use the following commands:

If the store has not been deployed:

kv-> change-policy -params systemPercent=100

If the store has been deployed:

kv-> plan change-parameters -service sn1 -wait -params systemPercent=100

Note:

The Replication Node (RN) does not re-start by default. You need to re-start the RN (or RNs if the capacity is greater than 1) by using the following commands:

kv-> plan stop-service -all-rns -wait;
Executed plan 7, waiting for completion...
Plan 7 ended successfully
kv-> plan start-service -all-rns -wait;
Executed plan 8, waiting for completion...
Plan 8 ended successfully

This yields:

A heap size of 32 GB and in-memory cache size of 22.4 GB, JVM overhead memory size of 8GB. Nothing has changed from the default configuration. ·
The system memory is 40 GB. System memory is 100% of what remains when the heap and JVM overhead are taken away, as follows: 80 GB (total available) - 32 GB (heap size) - 8 GB (JVM overhead) = 40 GB for the operating system and file system cache.

Figure E-2 SN Memory Without Off-heap Cache