Oracle Berkeley DB Java Edition 12c R1 Change Log

Library Version 12.1.6.0, Release 6.0.11

The open source license for JE has been changed. The open source license for JE 6.0 is the GNU AFFERO GENERAL PUBLIC LICENSE. See the LICENSE file for details. Commercial licensees are not impacted by this change.

The Java compatibility requirement for JE has changed from Java SE 6 to 7. JE is compatible with Java SE 7 and later, and has been tested and certified against Oracle JDK 7u51 and IBM J9 7.0. We encourage you to upgrade to the latest Java releases to take advantage of the latest bug fixes and performance improvements.

Log File On-Disk Format Changes:

JE 6.0.11 has moved to on-disk file format 9.

The change is forward compatible in that JE files created with release 5.0 and earlier can be read when opened with JE 6.0 or later. The change is not backward compatible in that files created with JE 6.0 or later cannot be read by earlier releases. Note that if an existing environment is opened read/write, a new log file is written by JE 6.0 or later, and the environment can no longer be read by earlier releases.

Upgrading from JE 5.0 or earlier

In addition to the file format change, a change was made involving partial Btree and duplicate comparators. Partial comparators are an advanced feature that few applications use. As of JE 6.0, using partial comparators is not recommended. Applications that do use partial comparators must change their comparator classes to implement the new PartialComparator tag interface, before running the application with JE 6. Failure to do so may cause incorrect behavior during transaction aborts. See the PartialComparator javadoc for more information.

Upgrading from JE 4.1 or earlier

There are two important notes about the file format change in JE 5.0.
  1. The file format change enabled significant improvements in operation performance, memory and disk footprint, and concurrency of databases with duplicate keys. Due to these changes, an upgrade utility must be run before opening an environment with this release, if the environment was created using JE 4.1 or earlier. See the Upgrade Procedure below for more information.
  2. An application which uses JE replication may not upgrade directly from JE 4.0 to JE 5.0 or later. Instead, the upgrade must be done from JE 4.0 to JE 4.1 and then to JE 5.0 or later. Applications already at JE 4.1 are not affected. Upgrade guidance can be found in the new chapter, "Upgrading a JE Replication Group", in the "Getting Started with BDB JE High Availability" guide.
Due to the format changes in JE 5, a special utility program must be run for an environment created with JE 4.1 or earlier, prior to opening the environment with JE 5.0 or later. The utility program is part of JE 4.1. JE 4.1.20, or a later version of JE 4.1, must be used.

One of two utility programs must be used, which are available in the release package for JE 4.1.20, or a later release of JE 4.1. If you are currently running a release earlier than JE 4.1.20, then you must download the latest JE 4.1 release package in order to run these utilities.

The steps for upgrading are as follows.

  1. Stop the application using BDB JE.
  2. Run the DbPreUpgrade_4_1 or DbRepPreUpgrade_4_1 utility. If you are using a regular non-replicated Environment:
        java -jar je-4.1.20.jar DbPreUpgrade_4_1 -h <dir>
    If you are using a JE ReplicatedEnvironment:
        java -jar je-4.1.20.jar DbRepPreUpgrade_4_1
             -h <dir>
             -groupName <group name>
             -nodeName <node name>
             -nodeHostPort <host:port>
  3. Finally, start the application using the current JE 5.0 (or later) release of BDB JE.

The second step -- running the utility program -- does not perform data conversion. This step simply performs a special checkpoint to prepare the environment for upgrade. It should take no longer than an ordinary startup and shutdown.

During the last step -- when the application opens the JE environment using the current release (JE 5 or later) -- all databases configured for duplicates will automatically be converted before the Environment or ReplicatedEnvironment constructor returns. Note that a database might be explicitly configured for duplicates using DatabaseConfig.setSortedDuplicates(true), or implicitly configured for duplicates by using a DPL MANY_TO_XXX relationship (Relationship.MANY_TO_ONE or Relationship.MANY_TO_MANY).

The duplicate database conversion only rewrites internal nodes in the Btree, not leaf nodes. In a test with a 500 MB cache, conversion of a 10 million record data set (8 byte key and data) took between 1.5 and 6.5 minutes, depending on number of duplicates per key. The high end of this range is when 10 duplicates per key were used; the low end is with 1 million duplicates per key.

To make the duplicate database conversion predictable during deployment, users should measure the conversion time on a non-production system before upgrading a deployed system. When duplicates are converted, the Btree internal nodes are preloaded into the JE cache. A new configuration option, EnvironmentConfig.ENV_DUP_CONVERT_PRELOAD_ALL, can be set to false to optimize this process if the cache is not large enough to hold the internal nodes for all databases. For more information, see the javadoc for this property.

If an application has no databases configured for duplicates, then the last step simply opens the JE environment normally, and no data conversion is performed.

If the user fails to run the DbPreUpgrade_4_1 or DbRepPreUpgrade_4_1 utility program before opening an environment with JE 5 or later for the first time, an exception such as the following will normally be thrown by the Environment or ReplicatedEnvironment constructor:

  com.sleepycat.je.EnvironmentFailureException: (JE 6.0.1) JE 4.1 duplicate DB
  entries were found in the recovery interval. Before upgrading to JE 5.0, the
  following utility must be run using JE 4.1 (4.1.20 or later):
  DbPreUpgrade_4_1.  See the change log.
  UNEXPECTED_STATE: Unexpected internal state, may have side effects.
    at com.sleepycat.je.EnvironmentFailureException.unexpectedState(EnvironmentFailureException.java:376)
    at com.sleepycat.je.recovery.RecoveryManager.checkLogVersion8UpgradeViolations(RecoveryManager.java:2694)
    at com.sleepycat.je.recovery.RecoveryManager.buildTree(RecoveryManager.java:549)
    at com.sleepycat.je.recovery.RecoveryManager.recover(RecoveryManager.java:198)
    at com.sleepycat.je.dbi.EnvironmentImpl.finishInit(EnvironmentImpl.java:610)
    ...  

If the user fails to run the DbPreUpgrade_4_1 or DbRepPreUpgrade_4_1 utility program, but no exception is thrown when the environment is opened with JE 5 or later, this is probably because the application performed an Environment.sync before last closing the environment with JE 4.1 or earlier, and nothing else happened to be written (by the application or JE background threads) after the sync operation. In this case, running the upgrade utility is not necessary.


Changes in 6.0.11

  1. Added support in JE HA for the new SECONDARY node type. SECONDARY nodes can only be replicas, not masters, and do not participate in either elections or durability decisions. SECONDARY nodes can be used to increase the available number of read replicas without changing the election or durability quorum of the group, and without requiring communication with the secondaries during master elections or transaction commits.

    Changes include adding the NodeType.SECONDARY enumeration constant, and the ReplicationGroup.getSecondaryNodes and ReplicationGroup.getDataNodes methods. [#22482] (6.0.1)


  2. Made improvements to internal latching to allow interrupting threads that are waiting on latches, cause a timeouts when a latch deadlock occurs, and enable latch instrumentation via system properties. Note that latching is not related to transactional locking and latches are intended to be held for very short periods. [#22993] (6.0.3)

  3. The following log cleaner configuration parameters in the EnvironmentConfig class have been deprecated and are no longer used. If configured, they will be silently ignored. Lazy and proactive migration are no longer supported due to negative impacts on eviction, checkpointing and Btree splits. If a persistent log cleaner backlog occurs, the recommended solution is to configure additional cleaner threads. [#23070] (6.0.3)

  4. When using secondary databases and DPL secondary indexes, the locking order for reads via a secondary has been changed to reduce the possibility of deadlocks. This optimization does not apply when the serializable isolation mode is used, and does not apply to the JoinCursor. [#22368] (6.0.4)

  5. Improved Btree cache usage by caching a BIN-delta -- the partial form of a BIN containing only the dirty entries -- in preference to logging it and then evicting the entire BIN. This reduces disk reads if CRUD operations are performed on the BIN before the entire BIN is evicted, because only one BIN fetch rather than two is needed. Disk writes are also reduced to some degree. The performance improvement applies only when BINs are being evicted from cache. The improvement is signficant when CRUD operations address a non-random subset of the keys in the data set.

    As part of the performance improvement work, the following statistics were added.

    In addition, the EnvironmentConfig.TREE_MAX_DELTA param has been deprecated. As of JE 5.0, the benefit from logging BIN-deltas is unrelated to the number of deltas that have been logged since the last full BIN. To configure BIN-delta logging, use EnvironmentConfig.TREE_BIN_DELTA.

    [#22662] (6.0.5)


  6. An optimization for Databases with sorted duplicates configured has been made to improve log cleaning performance. Records in duplicates databases need no longer be tracked or processed by the log cleaner, which reduces cleaning costs significantly when duplicates databases are used for a significant portion of a data set, for example, as secondary index databases.

    As described under 'Upgrading from JE 5.0 or earlier' at the top of this document, to support this cleaner optimization a change was made involving partial Btree and duplicate comparators. Partial comparators are an advanced feature that few applications use. As of JE 6.0, using partial comparators is not recommended. Applications that do use partial comparators must now change their comparator classes to implement the new PartialComparator tag interface, before running the application with JE 6. Failure to do so may cause incorrect behavior during transaction aborts. See the PartialComparator javadoc for more information.

    [#22864] (6.0.5)


  7. Fixed a bug that sometimes resulted in an uncommitted record deletion performed in one transaction to be visible (result in a NOTFOUND result) to an operation performed in another transaction. This bug applies to the use of Database.delete and PrimaryIndex.delete. It does not apply to the use of SecondaryDatabase.delete, SecondaryIndex.delete, or the use of a cursor to perform a deletion. Note that this problem is distinct from a similar bug that was fixed in JE 5.0.98 ([#22892]).

    [#23132] (6.0.5)


  8. Modified the algorithm that protects cleaned log files from deletion to consider the relative cost of replication replay versus network restore, as well as available disk space. When JE HA decides whether to delete cleaned log files, it uses information it stores about the progress of replication replay for each electable replica to retain useful log files even if the replicas are offline, subject to the ReplicationConfig.REP_STREAM_TIMEOUT parameter. The system does not store information about replication progress for secondary replicas, though, so a different approach has been added.

    The modified algorithm estimates the costs of replication replay and network restore, and protects log files from deletion that could be used for replay if there is sufficient disk space and replay would be less expensive than network restore. These computations apply to all replicas, but are particularly useful for secondary replicas, for which log files will not otherwise be retained if the replicas become temporarily unreachable. Note that disk space calculations are only performed when running with Java 7 or later.

    Two new ReplicationConfig parameters were added:

    [#22575] (6.0.5)


  9. An improvement was made to the calculation of log utilization to avoid under-cleaning or over-cleaning. For example, when log utilization was estimated to be lower than actual utilization, unnecessary over-cleaning would occur, which could reduce performance. Or when log utilization was estimated to be higher than actual utilization, under-cleaning would prevent reclaiming unused disk space.

    To prevent these problems, the size of each logged record is now stored in the Btree BINs (bottom internal nodes), so that utilization can be calculated correctly during record updates and deletions, while still avoiding a fetch of the old version of the record. With this change, the utilization adjustment facility in the log cleaner, which attempted to compensate for this problem by estimating utilization, is no longer needed by most applications.

    Therefore the EnvironmentConfig.CLEANER_ADJUST_UTILIZATION parameter is now false by default rather than true, and will be disabled completely in a future version of JE. For more information, see the javadoc for this parameter.

    [#22275] (6.0.7)


  10. The helper hosts parameter used in JE HA replication is now mutable. Accordingly, the set/getHelperHosts() methods and the HELPER_HOST definition in com.sleepycat.je.rep.ReplicationConfig have been moved to their parent class, ReplicationMutableConfig. The change is fully link and source compatible. [#22753] (6.0.7)

  11. Improved the performance of eviction by removing a bottleneck that was causing thread contention. Previously, for workloads with heavy eviction, threads were often waiting on a mutex in the TargetSelector.selectIN method. This impacted not only JE's dedicated background threads, but also application threads that were participating in critical eviction. A new approach is used that dramatically reduces thread contention and increases performance (compared to JE 5 and earlier) for such workloads.

    In addition, the new eviction approach implements a more accurate LRU which ensures that dirty nodes are evicted last and thereby reduces unnecessary logging.

    As part of this change, the following configuration parameters were deprecated and are ignored by JE:

        EnvironmentConfig.EVICTOR_NODES_PER_SCAN
        EnvironmentConfig.EVICTOR_LRU_ONLY
    
    And the following configuration parameter was added:
        EnvironmentConfig.EVICTOR_N_LRU_LISTS
    
    [#23063] (6.0.7)

  12. A change was made involving the charset for internal text (messages) that appear in the JE log (.jdb files). Previously, the default JVM charset was used. When dumping the log with DbPrintLog (e.g., for debugging purposes), if the default JVM charset was different than the one at the time the log was written, the text messages would be garbled. For example, this occurred when the log was written with an EBCDIC charset and then dumped with a UTF8 charset. This has been fixed by always writing and reading text in the UTF8 charset. [#15296] (6.0.8)

  13. A new HA configuration parameter: com.sleepycat.je.rep.ReplicationConfig.BIND_INADDR_ANY was added. This parameter permits binding of the port used by HA to all the local interfaces on the host. The javadoc associated with this configuration parameter provides further details. [#23437] (6.0.9)

  14. Fix a bug that could under rare conditions, primarily frequent failovers, cause the following exception in an HA environment.
    Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.97)
    node2(2):foo\node2 Read invisible log entry at 0x0/0xcb776
    hdr type="INS_LN_TX/8" vlsn v="19,373" isReplicated="1" isInvisible="1"
    prev="0xcb74c" size="17" cksum="2626620732"
    LOG_INTEGRITY: Log information is incorrect, problem is likely persistent.
    fetchTarget of 0x0/0xcb776 parent IN=29 IN class=com.sleepycat.je.tree.BIN
    lastFullVersion=0x0/0xf154c lastLoggedVersion=0x0/0xf588e
    parent.getDirty()=true state=3
    at com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:1054)
    at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:906)
    at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:867)
    at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1427)
    at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1250)
    at com.sleepycat.je.recovery.RecoveryManager.undo(RecoveryManager.java:2415)
    at com.sleepycat.je.recovery.RecoveryManager.rollbackUndo(RecoveryManager.java:2268)
    ...
    
    [#22848] (6.0.10)

  15. EntityStore.close has been changed to fix a bug that caused a memory leak when the Database could not be closed, for example, if it had open cursors. The javadoc for this method was also updated to warn that it must be called to avoid memory leaks, even when the Environment is invalid. [#23462] (6.0.10)