com.sleepycat.je.util

Class DbCacheSize

java.lang.Object

com.sleepycat.je.util.DbCacheSize

public class DbCacheSize
extends java.lang.Object

Estimates the in-memory cache size needed to hold a specified data set. To get an estimate of the in-memory footprint for a given database, specify the number of records and database characteristics and DbCacheSize will return an estimate of the cache size required for holding the database in memory. Based on this information a JE main cache size can be chosen and then configured using EnvironmentMutableConfig.setCacheSize(long) or using the EnvironmentConfig.MAX_MEMORY property. An off-heap cache may also be optionally configured using EnvironmentMutableConfig.setOffHeapCacheSize(long) or using the EnvironmentConfig.MAX_OFF_HEAP_MEMORY property.

Importance of the JE Cache

The JE cache is not an optional cache. It is used to hold the metadata for accessing JE data. In fact the JE cache size is probably the most critical factor to JE performance, since Btree nodes will have to be fetched during a database read or write operation if they are not in cache. During a single read or write operation, at each level of the Btree that a fetch is necessary, an IO may be necessary at a different disk location for each fetch. In addition, if internal nodes (INs) are not in cache, then write operations will cause additional copies of the INs to be written to storage, as modified INs are moved out of the cache to make room for other parts of the Btree during subsequent operations. This additional fetching and writing means that sizing the cache too small to hold the INs will result in lower operation performance.

For best performance, all Btree nodes should fit in the JE cache, including leaf nodes (LNs), which hold the record data, and INs, which hold record keys and other metadata. However, because system memory is limited, it is sometimes necessary to size the cache to hold all or at least most INs, but not the LNs. This utility estimates the size necessary to hold only INs, and the size to hold INs and LNs.

In addition, a common problem with large caches is that Java GC overhead can become significant. When a Btree node is evicted from the JE main cache based on JE's LRU algorithm, typically the node will have been resident in the JVM heap for an extended period of time, and will be expensive to GC. Therefore, when most or all LNs do not fit in the main cache, using CacheMode.EVICT_LN can be beneficial to reduce the Java GC cost of collecting the LNs as they are moved out of the main cache. With EVICT_LN, the LNs only reside in the JVM heap for a short period and are cheap to collect. A recommended approach is to size the JE main cache to hold only INs, and size the Java heap to hold that amount plus the amount needed for GC working space and application objects, leaving any additional memory for use by the file system cache or the off-heap cache. Tests show this approach results in lower GC overhead and more predictable latency.

Another issue is that 64-bit JVMs store object references using less space when the heap size is slightly less than 32GiB. When the heap size is 32GiB or more, object references are larger and less data can be cached per GiB of memory. This JVM feature is enabled with the Compressed Oops (-XX:+UseCompressedOops) option, although in modern JVMs it is on by default. Because of this factor, and because Java GC overhead is usually higher with larger heaps, a maximum heap size slightly less than 32GiB is recommended, along with Compressed Oops option.

Of course, the JE main cache size must be less than the heap size since the main cache is stored in the heap. In fact, around 30% of free space should normally be reserved in the heap for use by Java GC, to avoid high GC overheads. For example, if the application uses roughly 2GiB of the heap, then with a 32GiB heap the JE main cache should normally be no more than 20GiB.

As of JE 6.4, an optional off-heap cache may be configured in addition to the main JE cache. See EnvironmentMutableConfig.setOffHeapCacheSize(long) for information about the trade-offs in using an off-heap cache. When the -offheap argument is specified, this utility displays sizing information for both the main and off-heap caches. The portion of the data set that fits in the main cache, and the off-heap size needed to hold the rest of the data set, will be shown. The main cache size can be specified with the -maincache argument, or is implied to be the amount needed to hold all internal nodes if this argument is omitted. Omitting this argument is appropriate when CacheMode.EVICT_LN is used, since only internal nodes will be stored in the main cache.

To reduce Java GC overhead, sometimes a small main cache is used along with an off-heap cache. Note that it is important that the size the main cache is at least large enough to hold all the upper INs (the INs at level 2 and above). This is because the off-heap cache does not contain upper INs, it only contains LNs and bottom internal nodes (BINs). When a level 2 IN is evicted from the main cache, its children (BINs and LNs) in the off-heap cache, if any, must also be evicted, which can be undesirable, especially if the off-heap cache is not full. This utility displays the main cache size needed to hold all upper INs, and displays a warning if this is smaller than the main cache size specified.

Estimating the JE Cache Size

Estimating JE in-memory sizes is not straightforward for several reasons. There is some fixed overhead for each Btree internal node, so fanout (maximum number of child entries per parent node) and degree of node sparseness impacts memory consumption. In addition, JE uses various compact in-memory representations that depend on key sizes, data sizes, key prefixing, how many child nodes are resident, etc. The physical proximity of node children also allows compaction of child physical address values.

Therefore, when running this utility it is important to specify all EnvironmentConfig and DatabaseConfig settings that will be used in a production system. The EnvironmentConfig settings are specified by command line options for each property, using the same names as the EnvironmentConfig parameter name values. For example, EnvironmentConfig.LOG_FILE_MAX, which influences the amount of memory used to store physical record addresses, can be specified on the command line as:

-je.log.fileMax LENGTH

To be sure that this utility takes into account all relevant settings, especially as the utility is enhanced in future versions, it is best to specify all EnvironmentConfig settings used by the application.

The DatabaseConfig settings are specified using command line options defined by this utility.

• -nodemax ENTRIES corresponds to DatabaseConfig.setNodeMaxEntries(int).
• -duplicates corresponds to passing true to DatabaseConfig.setSortedDuplicates(boolean). Note that duplicates are configured for DPL MANY_TO_ONE and MANY_TO_MANY secondary indices.
• -keyprefix LENGTH corresponds to passing true DatabaseConfig.setKeyPrefixing(boolean). Note that key prefixing is always used when duplicates are configured.

This utility estimates the JE cache size by creating an in-memory Environment and Database. In addition to the size of the Database, the minimum overhead for the Environment is output. The Environment overhead shown is likely to be smaller than actually needed because it doesn't take into account use of memory by JE daemon threads (cleaner, checkpointer, etc) the memory used for locks that are held by application operations and transactions, the memory for HA network connections, etc. An additional amount should be added to account for these factors.

This utility estimates the cache size for a single JE Database, or a logical table spread across multiple databases (as in the case of Oracle NoSQL DB, for example). To estimate the size for multiple databases/tables with different configuration parameters or different key and data sizes, run this utility for each database/table and sum the sizes. If you are summing multiple runs for multiple databases/tables that are opened in a single Environment, the overhead size for the Environment should only be added once.

In some applications with databases/tables having variable key and data sizes, it may be difficult to determine the key and data size input parameters for this utility. If a representative data set can be created, one approach is to use the DbPrintLog utility with the -S option to find the average key and data size for all databases/tables, and use these values as input parameters, as if there were only a single database/tables. With this approach, it is important that the DatabaseConfig parameters are the same, or at least similar, for all databases/tables.

Key Prefixing and Compaction

Key prefixing deserves special consideration. It can significantly reduce the size of the cache and is generally recommended; however, the benefit can be difficult to predict. Key prefixing, in turn, impacts the benefits of key compaction, and the use of the EnvironmentConfig.TREE_COMPACT_MAX_KEY_LENGTH parameter.

For a given data set, the impact of key prefixing is determined by how many leading bytes are in common for the keys in a single bottom internal node (BIN). For example, if keys are assigned sequentially as long (8 byte) integers, and the maximum entries per node is 128 (the default value) then 6 or 7 of the 8 bytes of the key will have a common prefix in each BIN. Of course, when records are deleted, the number of prefixed bytes may be reduced because the range of key values in a BIN will be larger. For this example we will assume that, on average, 5 bytes in each BIN are a common prefix leaving 3 bytes per key that are unprefixed.

Key compaction is applied when the number of unprefixed bytes is less than a configured value; see EnvironmentConfig.TREE_COMPACT_MAX_KEY_LENGTH. In the example, the 3 unprefixed bytes per key is less than the default used for key compaction (16 bytes). This means that each key will use 16 bytes of memory, in addition to the amount used for the prefix for each BIN. The per-key overhead could be reduced by changing the TREE_COMPACT_MAX_KEY_LENGTH parameter to a smaller value, but care should be taken to ensure the compaction will be effective as keys are inserted and deleted over time.

Because key prefixing depends so much on the application key format and the way keys are assigned, the number of expected prefix bytes must be estimated by the user and specified to DbCacheSize using the -keyprefix argument.

Key Prefixing and Duplicates

When duplicates are configured for a Database (including DPL MANY_TO_ONE and MANY_TO_MANY secondary indices), key prefixing is always used. This is because the internal key in a duplicates database BIN is formed by concatenating the user-specified key and data. In secondary databases with duplicates configured, the data is the primary key, so the internal key is the concatenation of the secondary key and the primary key.

Key prefixing is always used for duplicates databases because prefixing is necessary to store keys efficiently. When the number of duplicates per unique user-specified key is more than the number of entries per BIN, the entire user-specified key will be the common prefix.

For example, a database that stores user information may use email address as the primary key and zip code as a secondary key. The secondary index database will be a duplicates database, and the internal key stored in the BINs will be a two part key containing zip code followed by email address. If on average there are more users per zip code than the number of entries in a BIN, then the key prefix will normally be at least as long as the zip code key. If there are less (more than one zip code appears in each BIN), then the prefix will be shorter than the zip code key.

It is also possible for the key prefix to be larger than the secondary key. If for one secondary key value (one zip code) there are a large number of primary keys (email addresses), then a single BIN may contain concatenated keys that all have the same secondary key (same zip code) and have primary keys (email addresses) that all have some number of prefix bytes in common. Therefore, when duplicates are specified it is possible to specify a prefix size that is larger than the key size.

Small Data Sizes and Embedded LNs

Another special data representation involves small data sizes. When the data size of a record is less than or equal to EnvironmentConfig.TREE_MAX_EMBEDDED_LN (16 bytes, by default), the data is stored (embedded) in the BIN, and the LN is not stored in cache at all. This increases the size needed to hold all INs in cache, but it decreases the size needed to hold the complete data set. If the data size specified when running this utility is less than or equal to TREE_MAX_EMBEDDED_LN, the size displayed for holding INs only will be the same as the size displayed for holdings INs and LNs.

See EnvironmentConfig.TREE_MAX_EMBEDDED_LN for information about the trade-offs in using the embedded LNs feature.

Record Versions and Oracle NoSQL Database

This note applies only to when JE is used with Oracle NoSQL DB. In Oracle NoSQL DB, an internal JE environment configuration parameter is always used: -je.rep.preserveRecordVersion true. This allows using record versions in operations such as "put if version", "delete if version", etc. This feature performs best when the cache is sized large enough to hold the record versions.

When using JE with Oracle NoSQL DB, always add -je.rep.preserveRecordVersion true to the command line. This ensures that the cache sizes calculated are correct, and also outputs an additional line showing how much memory is required to hold the internal nodes and record versions (but not the leaf nodes). This is the minimum recommended size when the "... if version" operations are used.

Running the DbCacheSize utility

Usage:

 java { com.sleepycat.je.util.DbCacheSize |
        -jar je-<version>.jar DbCacheSize }
  -records COUNT
      # Total records (key/data pairs); required
  -key BYTES
      # Average key bytes per record; required
  [-data BYTES]
      # Average data bytes per record; if omitted no leaf
      # node sizes are included in the output; required with
      # -duplicates, and specifies the primary key length
  [-offheap]
      # Indicates that an off-heap cache will be used.
  [-maincache BYTES]
      # The size of the main cache (in the JVM heap).
      # The size of the off-heap cache displayed is the
      # additional amount needed to hold the data set.
      # If omitted, the main cache size is implied to
      # be the amount needed to hold all internal nodes.
      # Ignored if -offheap is not also specified.
  [-keyprefix BYTES]
      # Expected size of the prefix for the keys in each
      # BIN; default: key prefixing is not configured;
      # required with -duplicates
  [-nodemax ENTRIES]
      # Number of entries per Btree node; default: 128
  [-orderedinsertion]
      # Assume ordered insertions and no deletions, so BINs
      # are 100% full; default: unordered insertions and/or
      # deletions, BINs are 70% full
  [-duplicates]
      # Indicates that sorted duplicates are used, including
      # MANY_TO_ONE and MANY_TO_MANY secondary indices;
      # default: false
  [-ttl]
      # Indicates that TTL is used; default: false
  [-replicated]
      # Use a ReplicatedEnvironment; default: false
  [-ENV_PARAM_NAME VALUE]...
      # Any number of EnvironmentConfig parameters and
      # ReplicationConfig parameters (if -replicated)
  [-btreeinfo]
      # Outputs additional Btree information
  [-outputproperties]
      # Writes Java properties file to System.out
 

You should run DbCacheSize on the same target platform and JVM for which you are sizing the cache, as cache sizes will vary. You may also need to specify -d32 or -d64 depending on your target, if the default JVM mode is not the same as the mode to be used in production.

To take full advantage of JE cache memory, it is strongly recommended that compressed oops (-XX:+UseCompressedOops) is specified when a 64-bit JVM is used and the maximum heap size is less than 32 GB. As described in the referenced documentation, compressed oops is sometimes the default JVM mode even when it is not explicitly specified in the Java command. However, if compressed oops is desired then it must be explicitly specified in the Java command when running DbCacheSize or a JE application. If it is not explicitly specified then JE will not aware of it, even if it is the JVM default setting, and will not take it into account when calculating cache memory sizes.

For example:

 $ java -jar je-X.Y.Z.jar DbCacheSize -records 554719 -key 16 -data 100

  === Environment Cache Overhead ===

  3,157,213 minimum bytes

 To account for JE daemon operation, record locks, HA network connections, etc,
 a larger amount is needed in practice.

  === Database Cache Size ===

  Number of Bytes  Description
  ---------------  -----------
       23,933,736  Internal nodes only
      107,206,616  Internal nodes and leaf nodes
 

This indicates that the minimum memory size to hold only the internal nodes of the Database Btree is approximately 24MB. The maximum size to hold the entire database, both internal nodes and data records, is approximately 107MB. To either of these amounts, at least 3MB (plus more for locks and daemons) should be added to account for the environment overhead.

The following example adds the use of an off-heap cache, where the main cache size is specified to be 30MB.

 $ java -jar je-X.Y.Z.jar DbCacheSize -records 554719 -key 16 -data 100 \
      -offheap -maincache 30000000

  === Environment Cache Overhead ===

  5,205,309 minimum bytes

 To account for JE daemon operation, record locks, HA network connections, etc,
 a larger amount is needed in practice.

  === Database Cache Size ===

  Number of Bytes  Description
  ---------------  -----------
       23,933,736  Internal nodes only: MAIN cache
                0  Internal nodes only: OFF-HEAP cache
       24,794,691  Internal nodes and leaf nodes: MAIN cache
       70,463,604  Internal nodes and leaf nodes: OFF-HEAP cache
 

There are several things of interest in the output.

• The environment overhead is larger because of memory used for the off-heap LRU.
• To cache only internal nodes, an off-heap cache is not needed since the internal nodes take around 24MB, which when added to the 5MB overhead is less than the 30MB main cache specified. This is why the number of bytes on the second line is zero.
• To cache all nodes, the main cache size specified should be used (25MB added to the 5MB overhead is 30MB), and an off-heap cache of around 71MB should be configured.

Output Properties

When -outputproperties is specified, a list of properties in Java properties file format will be written to System.out, instead of the output shown above. The properties and their meanings are listed below.

• The following properties are always output (except allNodes, see below). They describe the estimated size of the main cache.
  • overhead: The environment overhead, as shown under Environment Cache Overhead above.
  • internalNodes: The Btree size in the main cache for holding the internal nodes. This is the "Internal nodes only" line above (followed by "MAIN cache" when -offheap is specified).
  • internalNodesAndVersions: The Btree size needed to hold the internal nodes and record versions in the main cache. This value is zero when -offheap is specified; currently JE does not cache record versions off-heap unless their associated LNs are also cached off-heap, so there is no way to calculate this property.
  • allNodes: The Btree size in the main cache needed to hold all nodes. This is the "Internal nodes and leaf nodes" line above (followed by "MAIN cache" when -offheap is specified). This property is not output unless -data is specified.
• The following properties are output only when -offheap is specified. They describe the estimated size of the off-heap cache.
  • minMainCache: The minimum size of the main cache needed to hold all upper INs. When the -maincache value specified is less than this minimum, not all internal nodes can be cached. See the discussion further above.
  • offHeapInternalNodes: The size of the off-heap cache needed to hold the internal nodes. This is the "Internal nodes only: OFF_HEAP cache" line above.
  • offHeapAllNodes: The size of the off-heap cache needed to hold all nodes. This is the "Internal nodes and leaf nodes: OFF_HEAP cache" line above. This property is not output unless -data is specified.
• The following properties are deprecated but are output for compatibility with earlier releases.
  • minInternalNodes, maxInternalNodes, minAllNodes, and (when -data is specified) maxAllNodes

See Also:

EnvironmentMutableConfig.setCacheSize(long), EnvironmentMutableConfig.setOffHeapCacheSize(long), CacheMode, Cache Statistics: Sizing

Method Summary

Modifier and Type	Method and Description
static void	main(java.lang.String[] args) Runs DbCacheSize as a command line utility.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

main

public static void main(java.lang.String[] args)
                 throws java.lang.Throwable

Runs DbCacheSize as a command line utility. For command usage, see class description.

Throws:

java.lang.Throwable